2024-01-11T04:48:55.494 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-11T04:48:55.494 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279 --verbose --owner scheduled_yuriw@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/orig.config.yaml 2024-01-11T04:48:55.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T04:48:55.640 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279 branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7512279' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 419 sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNh5DBjwgYTSwL/3YloYZJSjDOD+LxmDNsWW17id/sGjVny/f2XlzmuPkW+zwV6hQbcSkEIKHIPSFEzoPJI+QRM= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJxRwirZkrqx5TImKRLdwq1OP0U4I4EXXu+V8JheRl7rQwhiWlPFk5nAsBGmKYad7QJ2R5mPWpMHZQHN+mgit8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-10_19:20:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-11T04:48:55.641 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa; will attempt to use it 2024-01-11T04:48:55.641 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks 2024-01-11T04:48:55.642 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-11T04:48:55.643 INFO:teuthology.task.internal:Checking packages... 2024-01-11T04:48:55.668 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4bcedb9fc11bed1fd6fd0c4fd59187401f309680' 2024-01-11T04:48:55.668 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T04:48:55.668 INFO:teuthology.packaging:ref: None 2024-01-11T04:48:55.668 INFO:teuthology.packaging:tag: None 2024-01-11T04:48:55.668 INFO:teuthology.packaging:branch: wip-vshankar-testing1-quincy-2024-01-10-2010-quincy 2024-01-11T04:48:55.669 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T04:48:55.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing1-quincy-2024-01-10-2010-quincy 2024-01-11T04:48:55.887 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-299.g4bcedb9f 2024-01-11T04:48:55.888 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-11T04:48:55.895 INFO:teuthology.task.internal:no buildpackages task found 2024-01-11T04:48:55.895 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-11T04:48:55.926 INFO:teuthology.task.internal:Saving configuration 2024-01-11T04:48:55.944 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-11T04:48:55.950 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T04:48:55.976 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 04:43:46.658862', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNh5DBjwgYTSwL/3YloYZJSjDOD+LxmDNsWW17id/sGjVny/f2XlzmuPkW+zwV6hQbcSkEIKHIPSFEzoPJI+QRM='} 2024-01-11T04:48:56.000 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 04:43:46.656854', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJxRwirZkrqx5TImKRLdwq1OP0U4I4EXXu+V8JheRl7rQwhiWlPFk5nAsBGmKYad7QJ2R5mPWpMHZQHN+mgit8='} 2024-01-11T04:48:56.001 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-11T04:48:56.007 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-11T04:48:56.007 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-11T04:48:56.007 INFO:teuthology.run_tasks:Running task console_log... 2024-01-11T04:48:56.092 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3839876ee0>, signals=[15]) 2024-01-11T04:48:56.092 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-11T04:48:56.099 INFO:teuthology.task.internal:Opening connections... 2024-01-11T04:48:56.099 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-01-11T04:48:56.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:48:56.174 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-01-11T04:48:56.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:48:56.247 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-11T04:48:56.254 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-01-11T04:48:56.277 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-01-11T04:48:56.277 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-01-11T04:48:56.338 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-01-11T04:48:56.338 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8" 2024-01-11T04:48:56.338 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-01-11T04:48:56.338 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-01-11T04:48:56.338 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8" 2024-01-11T04:48:56.338 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-11T04:48:56.339 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-11T04:48:56.339 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-01-11T04:48:56.359 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-01-11T04:48:56.381 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-01-11T04:48:56.382 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-01-11T04:48:56.443 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-01-11T04:48:56.443 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="8" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="8" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el8" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-01-11T04:48:56.444 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-11T04:48:56.445 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-11T04:48:56.445 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-11T04:48:56.445 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-01-11T04:48:56.467 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-11T04:48:56.476 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-11T04:48:56.509 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-11T04:48:56.510 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-01-11T04:48:56.512 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-01-11T04:48:56.532 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-11T04:48:56.537 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-11T04:48:56.537 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-01-11T04:48:56.574 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-01-11T04:48:56.609 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-11T04:48:56.665 INFO:teuthology.run_tasks:Running task kernel... 2024-01-11T04:48:56.680 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-11T04:48:56.680 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-11T04:48:56.681 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T04:48:56.681 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T04:48:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T04:48:56.701 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-01-11T04:48:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T04:48:56.703 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-01-11T04:48:56.760 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-489.el8.x86_64 2024-01-11T04:48:56.761 INFO:teuthology.task.kernel:Running kernel on smithi169: 4.18.0-489.el8.x86_64 2024-01-11T04:48:56.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-01-11T04:48:56.763 INFO:teuthology.orchestra.run.smithi067.stdout:4.18.0-489.el8.x86_64 2024-01-11T04:48:56.763 INFO:teuthology.task.kernel:Running kernel on smithi067: 4.18.0-489.el8.x86_64 2024-01-11T04:48:56.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-01-11T04:49:00.901 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - AppStream 23 MB/s | 35 MB 00:01 2024-01-11T04:49:00.927 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - AppStream 21 MB/s | 35 MB 00:01 2024-01-11T04:49:08.710 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - BaseOS 41 MB/s | 58 MB 00:01 2024-01-11T04:49:10.018 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - BaseOS 27 MB/s | 58 MB 00:02 2024-01-11T04:49:17.134 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras 104 kB/s | 18 kB 00:00 2024-01-11T04:49:17.568 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras common packages 28 kB/s | 6.9 kB 00:00 2024-01-11T04:49:17.928 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-01-11T04:49:19.094 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras 56 kB/s | 18 kB 00:00 2024-01-11T04:49:19.095 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-01-11T04:49:19.553 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 6.9 kB 00:00 2024-01-11T04:49:19.863 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-01-11T04:49:21.899 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2024-01-11T04:49:22.987 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 268 kB/s | 24 kB 00:00 2024-01-11T04:49:26.080 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 406 kB/s | 24 kB 00:00 2024-01-11T04:49:26.704 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-11T04:49:26.704 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-11T04:49:27.018 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T04:49:27.018 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:49:27.019 INFO:teuthology.orchestra.run.smithi169.stdout:Install 3 Packages 2024-01-11T04:49:27.020 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:49:27.020 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 90 M 2024-01-11T04:49:27.020 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 96 M 2024-01-11T04:49:27.020 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T04:49:27.663 INFO:teuthology.orchestra.run.smithi169.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 23 MB/s | 10 MB 00:00 2024-01-11T04:49:28.397 INFO:teuthology.orchestra.run.smithi169.stdout:(2/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 31 MB/s | 36 MB 00:01 2024-01-11T04:49:28.931 INFO:teuthology.orchestra.run.smithi169.stdout:(3/3): kernel-core-4.18.0-532.el8.x86_64.rpm 25 MB/s | 43 MB 00:01 2024-01-11T04:49:28.931 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:49:28.931 INFO:teuthology.orchestra.run.smithi169.stdout:Total 47 MB/s | 90 MB 00:01 2024-01-11T04:49:29.606 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T04:49:30.042 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T04:49:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T04:49:30.063 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-11T04:49:30.063 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-11T04:49:30.383 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T04:49:30.384 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:49:30.384 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-01-11T04:49:30.384 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:49:30.384 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T04:49:30.384 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-11T04:49:30.384 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout:Install 3 Packages 2024-01-11T04:49:30.385 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:49:30.386 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 90 M 2024-01-11T04:49:30.386 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 96 M 2024-01-11T04:49:30.386 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T04:49:30.973 INFO:teuthology.orchestra.run.smithi067.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 26 MB/s | 10 MB 00:00 2024-01-11T04:49:31.285 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T04:49:31.328 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T04:49:31.823 INFO:teuthology.orchestra.run.smithi067.stdout:(2/3): kernel-core-4.18.0-532.el8.x86_64.rpm 35 MB/s | 43 MB 00:01 2024-01-11T04:49:32.340 INFO:teuthology.orchestra.run.smithi067.stdout:(3/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 21 MB/s | 36 MB 00:01 2024-01-11T04:49:32.341 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:49:32.341 INFO:teuthology.orchestra.run.smithi067.stdout:Total 46 MB/s | 90 MB 00:01 2024-01-11T04:49:33.051 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T04:49:33.490 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T04:49:33.490 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T04:49:34.126 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T04:49:34.618 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-11T04:49:34.802 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T04:49:34.829 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T04:49:37.772 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T04:49:38.234 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-11T04:49:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-11T04:49:38.617 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-11T04:49:42.047 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-11T04:49:42.068 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-11T04:49:42.292 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:49:42.498 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-11T04:49:46.182 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-11T04:49:46.459 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:04.575 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:04.600 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:05.075 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:05.075 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-11T04:50:05.075 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-11T04:50:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:05.890 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:50:05.890 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T04:50:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-11T04:50:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-11T04:50:05.891 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:50:05.891 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T04:50:06.038 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T04:50:06.256 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-11T04:50:06.256 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-11T04:50:06.256 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-11T04:50:06.256 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-11T04:50:06.257 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-11T04:50:06.257 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-11T04:50:06.257 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-01-11T04:50:06.257 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T04:50:06.467 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-11T04:50:06.467 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-11T04:50:06.467 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.02 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 129.el8 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Group : System Environment/Base 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 9265586 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-01-11T04:50:06.501 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Relocations : (not relocatable) 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : CentOS BuildSystem 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-11T04:50:06.502 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-01-11T04:50:06.503 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:50:06.503 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-01-11T04:50:06.503 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-11T04:50:06.503 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 4.18.0-532.el8.x86_64 2024-01-11T04:50:06.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-11T04:50:07.909 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-01-11T04:50:10.018 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:10.044 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:10.095 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-01-11T04:50:10.097 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-01-11T04:50:10.130 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.UJcxSxfsVJ 2024-01-11T04:50:10.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UJcxSxfsVJ 2024-01-11T04:50:10.205 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.UJcxSxfsVJ 2024-01-11T04:50:10.363 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.UJcxSxfsVJ is 6KB 2024-01-11T04:50:10.400 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:10.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-11T04:50:10.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-11T04:50:10.410 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.UJcxSxfsVJ 2024-01-11T04:50:10.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-01-11T04:50:10.503 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-11T04:50:10.503 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-11T04:50:10.503 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-11T04:50:10.503 INFO:teuthology.orchestra.run.smithi169.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-11T04:50:10.504 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-11T04:50:10.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-01-11T04:50:11.259 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-11T04:50:11.259 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:50:11.260 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T04:50:11.260 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-11T04:50:11.260 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-11T04:50:11.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:50:11.260 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T04:50:11.436 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T04:50:11.661 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-11T04:50:11.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-11T04:50:11.661 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-11T04:50:11.661 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-11T04:50:11.662 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-11T04:50:11.662 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-11T04:50:11.662 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-01-11T04:50:11.662 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T04:50:11.880 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-11T04:50:11.881 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-11T04:50:11.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-01-11T04:50:11.915 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-01-11T04:50:11.915 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-01-11T04:50:11.915 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.02 2024-01-11T04:50:11.915 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 129.el8 2024-01-11T04:50:11.915 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-01-11T04:50:11.915 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Group : System Environment/Base 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 9265586 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Relocations : (not relocatable) 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : CentOS BuildSystem 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-01-11T04:50:11.916 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:50:11.917 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-01-11T04:50:11.918 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-11T04:50:11.918 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 4.18.0-532.el8.x86_64 2024-01-11T04:50:11.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-11T04:50:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-01-11T04:50:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-01-11T04:50:15.635 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-01-11T04:50:15.656 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.FYe9RqAzcG 2024-01-11T04:50:15.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FYe9RqAzcG 2024-01-11T04:50:15.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.FYe9RqAzcG 2024-01-11T04:50:15.879 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.FYe9RqAzcG is 6KB 2024-01-11T04:50:15.927 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.FYe9RqAzcG 2024-01-11T04:50:15.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-01-11T04:50:16.020 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-11T04:50:16.020 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-11T04:50:16.020 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-11T04:50:16.020 INFO:teuthology.orchestra.run.smithi067.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-11T04:50:16.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-11T04:50:16.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-01-11T04:50:40.716 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-11T04:50:40.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:50:40.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:50:46.184 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-11T04:50:46.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:50:46.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:50:59.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-01-11T04:51:04.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-01-11T04:51:08.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:51:08.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:51:11.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-01-11T04:51:13.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:51:13.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:51:16.704 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-01-11T04:51:23.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:51:23.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:51:28.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:51:28.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:51:41.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-01-11T04:51:47.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-01-11T04:51:56.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:51:56.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:51:57.148 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-01-11T04:51:57.423 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:51:57.423 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-532.el8.x86_64"... 2024-01-11T04:51:57.423 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-01-11T04:51:57.439 INFO:teuthology.orchestra.run.smithi169.stdout:4.18.0-532.el8.x86_64 2024-01-11T04:51:57.440 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-11T04:51:57.440 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T04:51:57.440 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-11T04:51:58.440 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-11T04:51:58.441 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-11T04:51:58.512 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-01-11T04:51:58.525 DEBUG:teuthology.parallel:result is None 2024-01-11T04:52:02.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:52:02.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:52:02.441 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-01-11T04:52:02.702 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:52:02.703 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-532.el8.x86_64"... 2024-01-11T04:52:02.703 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-01-11T04:52:02.720 INFO:teuthology.orchestra.run.smithi067.stdout:4.18.0-532.el8.x86_64 2024-01-11T04:52:02.721 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-11T04:52:02.721 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T04:52:02.721 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-11T04:52:03.722 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-11T04:52:03.722 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-11T04:52:03.789 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-01-11T04:52:03.807 DEBUG:teuthology.parallel:result is None 2024-01-11T04:52:03.807 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-11T04:52:03.814 INFO:teuthology.task.internal:Creating test directory... 2024-01-11T04:52:03.814 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T04:52:03.817 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T04:52:03.835 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-11T04:52:03.867 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-11T04:52:03.900 INFO:teuthology.task.internal:Creating archive directory... 2024-01-11T04:52:03.900 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T04:52:03.902 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T04:52:03.929 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-11T04:52:03.938 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-11T04:52:03.938 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T04:52:03.963 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T04:52:03.994 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T04:52:04.003 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T04:52:04.142 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T04:52:04.153 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T04:52:04.154 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-11T04:52:04.165 INFO:teuthology.task.internal:Configuring sudo... 2024-01-11T04:52:04.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T04:52:04.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T04:52:04.194 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-11T04:52:04.202 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-11T04:52:04.202 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T04:52:04.232 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T04:52:04.252 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T04:52:04.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T04:52:04.390 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T04:52:04.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T04:52:04.520 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T04:52:04.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T04:52:04.666 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T04:52:04.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T04:52:04.781 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T04:52:04.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T04:52:04.866 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T04:52:04.866 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T04:52:04.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-01-11T04:52:04.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-01-11T04:52:05.055 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T04:52:05.387 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T04:52:05.452 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-11T04:52:05.459 INFO:teuthology.task.internal:Starting timer... 2024-01-11T04:52:05.459 INFO:teuthology.run_tasks:Running task pcp... 2024-01-11T04:52:05.513 INFO:teuthology.run_tasks:Running task selinux... 2024-01-11T04:52:05.553 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-11T04:52:05.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-01-11T04:52:06.311 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-01-11T04:52:06.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-01-11T04:52:06.424 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-01-11T04:52:06.426 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-11T04:52:06.426 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-01-11T04:52:06.451 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-01-11T04:52:06.452 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-01-11T04:52:06.481 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-01-11T04:52:06.481 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-01-11T04:52:06.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T04:52:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T04:52:06.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T04:52:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T04:52:06.549 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-11T04:52:06.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-01-11T04:52:06.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-01-11T04:52:06.661 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-11T04:52:06.669 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-11T04:52:06.685 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-11T04:52:06.686 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-11T04:57:56.995 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-01-11T04:57:56.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:57:56.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:57:57.070 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-01-11T04:57:57.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-01-11T04:57:57.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:57:57.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T04:57:57.211 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-01-11T04:57:57.279 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-01-11T04:57:57.279 INFO:teuthology.run_tasks:Running task clock... 2024-01-11T04:57:57.290 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-11T04:57:57.290 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T04:57:57.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T04:57:57.293 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T04:57:57.293 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T04:57:57.323 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-11T04:57:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-11T04:57:57.351 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-11T04:57:57.365 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-11T04:57:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-01-11T04:57:57.414 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-01-11T04:57:57.429 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-11T04:57:57.436 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-01-11T04:57:57.442 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-11T04:57:57.447 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-01-11T04:57:57.461 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-11T04:57:57.474 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-11T04:57:57.509 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2024-01-11T04:57:57.513 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T04:57:57.513 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-01-11T04:57:57.513 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.513 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.513 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.513 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.541 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2024-01-11T04:57:57.544 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T04:57:57.544 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-01-11T04:57:57.545 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.545 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.545 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.545 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T04:57:57.546 INFO:teuthology.run_tasks:Running task pexec... 2024-01-11T04:57:57.555 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-11T04:57:57.555 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi067.front.sepia.ceph.com 2024-01-11T04:57:57.555 DEBUG:teuthology.orchestra.run.smithi067:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-11T04:57:57.556 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi169.front.sepia.ceph.com 2024-01-11T04:57:57.556 DEBUG:teuthology.orchestra.run.smithi169:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-11T04:57:58.472 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:32 ago on Thu 11 Jan 2024 04:56:26 AM UTC. 2024-01-11T04:57:58.475 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:25 ago on Thu 11 Jan 2024 04:56:33 AM UTC. 2024-01-11T04:58:00.669 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout:Resetting modules: 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T04:58:00.670 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:00.671 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:00.831 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T04:58:00.832 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:00.832 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-01-11T04:58:00.832 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:00.832 INFO:teuthology.orchestra.run.smithi067.stdout:Resetting modules: 2024-01-11T04:58:00.832 INFO:teuthology.orchestra.run.smithi067.stdout: container-tools 2024-01-11T04:58:00.833 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:00.833 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T04:58:00.833 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:00.833 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:00.936 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T04:58:01.085 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T04:58:01.894 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:35 ago on Thu 11 Jan 2024 04:56:26 AM UTC. 2024-01-11T04:58:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:28 ago on Thu 11 Jan 2024 04:56:33 AM UTC. 2024-01-11T04:58:04.059 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout:Installing group/module packages: 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-11T04:58:04.253 INFO:teuthology.orchestra.run.smithi169.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout:Installing module profiles: 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools/common 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:04.254 INFO:teuthology.orchestra.run.smithi169.stdout:Enabling module streams: 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout: container-tools rhel8 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 27 M 2024-01-11T04:58:04.255 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 89 M 2024-01-11T04:58:04.256 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T04:58:04.355 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout:Installing group/module packages: 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-11T04:58:04.562 INFO:teuthology.orchestra.run.smithi067.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout:Installing module profiles: 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: container-tools/common 2024-01-11T04:58:04.563 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling module streams: 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: container-tools rhel8 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:04.564 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 27 M 2024-01-11T04:58:04.565 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 89 M 2024-01-11T04:58:04.565 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T04:58:04.972 INFO:teuthology.orchestra.run.smithi169.stdout:[MIRROR] buildah-1.32.0-1.module_el8+712+4cd1bd69.x86_64.rpm: Status code: 500 for http://tx-mirror.tier.net/centos/8-stream/AppStream/x86_64/os/Packages/buildah-1.32.0-1.module_el8%2b712%2b4cd1bd69.x86_64.rpm (IP: 181.215.47.106) 2024-01-11T04:58:05.213 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.3 MB/s | 750 kB 00:00 2024-01-11T04:58:05.255 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.8 MB/s | 1.0 MB 00:00 2024-01-11T04:58:05.272 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 648 kB/s | 38 kB 00:00 2024-01-11T04:58:05.368 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.1 MB/s | 241 kB 00:00 2024-01-11T04:58:05.438 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.6 MB/s | 1.0 MB 00:00 2024-01-11T04:58:05.463 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 1.8 MB/s | 750 kB 00:00 2024-01-11T04:58:05.472 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2024-01-11T04:58:05.505 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 567 kB/s | 38 kB 00:00 2024-01-11T04:58:05.556 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 297 kB/s | 25 kB 00:00 2024-01-11T04:58:05.606 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.7 MB/s | 241 kB 00:00 2024-01-11T04:58:05.640 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-01-11T04:58:05.722 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.5 MB/s | 178 kB 00:00 2024-01-11T04:58:05.824 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 12 MB/s | 3.9 MB 00:00 2024-01-11T04:58:05.824 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] python3-pytoml-0.1.14-5.git7dea353.el8.noarch.rpm: Status code: 500 for http://tx-mirror.tier.net/centos/8-stream/AppStream/x86_64/os/Packages/python3-pytoml-0.1.14-5.git7dea353.el8.noarch.rpm (IP: 181.215.47.106) 2024-01-11T04:58:05.890 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-01-11T04:58:05.923 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 6.0 MB/s | 3.9 MB 00:00 2024-01-11T04:58:05.948 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-pytoml-0.1.14-5.git7dea353.el8. 110 kB/s | 25 kB 00:00 2024-01-11T04:58:06.349 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 6.6 MB/s | 9.6 MB 00:01 2024-01-11T04:58:06.416 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 5.1 MB/s | 2.5 MB 00:00 2024-01-11T04:58:06.491 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2024-01-11T04:58:06.591 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): udica-0.2.6-20.module_el8+712+4cd1bd69 217 kB/s | 53 kB 00:00 2024-01-11T04:58:06.624 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): skopeo-1.13.3-1.module_el8+712+4cd1bd69 11 MB/s | 8.1 MB 00:00 2024-01-11T04:58:06.674 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 3.5 MB/s | 2.5 MB 00:00 2024-01-11T04:58:06.691 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 787 kB/s | 53 kB 00:00 2024-01-11T04:58:06.758 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 490 kB/s | 41 kB 00:00 2024-01-11T04:58:06.808 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 7.0 MB/s | 8.1 MB 00:01 2024-01-11T04:58:06.809 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:58:06.809 INFO:teuthology.orchestra.run.smithi169.stdout:Total 10 MB/s | 27 MB 00:02 2024-01-11T04:58:06.967 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): buildah-1.32.0-1.module_el8+712+4cd1bd 5.0 MB/s | 9.6 MB 00:01 2024-01-11T04:58:06.967 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:58:06.967 INFO:teuthology.orchestra.run.smithi067.stdout:Total 11 MB/s | 27 MB 00:02 2024-01-11T04:58:07.075 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T04:58:07.096 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T04:58:07.096 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T04:58:07.242 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T04:58:07.264 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T04:58:07.264 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T04:58:07.354 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T04:58:07.357 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T04:58:07.531 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T04:58:07.534 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T04:58:08.045 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T04:58:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T04:58:08.277 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-11T04:58:08.388 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-11T04:58:08.439 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-11T04:58:08.540 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-11T04:58:08.643 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-11T04:58:08.710 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-11T04:58:08.872 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-11T04:58:08.992 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-11T04:58:09.440 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-11T04:58:09.543 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-11T04:58:10.392 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-11T04:58:10.585 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-11T04:58:10.617 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-11T04:58:10.794 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-11T04:58:10.855 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-11T04:58:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-11T04:58:11.110 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-11T04:58:11.245 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-11T04:58:11.518 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-11T04:58:11.649 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-11T04:58:12.423 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-11T04:58:12.571 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-11T04:58:12.660 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-11T04:58:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-11T04:58:12.846 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-11T04:58:13.097 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-11T04:58:13.279 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-11T04:58:13.279 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-11T04:58:13.279 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-11T04:58:13.279 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-11T04:58:13.280 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-11T04:58:13.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-11T04:58:13.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-11T04:58:14.121 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-11T04:58:14.121 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:14.121 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T04:58:14.121 INFO:teuthology.orchestra.run.smithi169.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.121 INFO:teuthology.orchestra.run.smithi169.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:14.122 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T04:58:14.363 DEBUG:teuthology.parallel:result is None 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-11T04:58:14.399 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-11T04:58:14.400 INFO:teuthology.orchestra.run.smithi067.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.400 INFO:teuthology.orchestra.run.smithi067.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-11T04:58:14.400 INFO:teuthology.orchestra.run.smithi067.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-11T04:58:14.400 INFO:teuthology.orchestra.run.smithi067.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-11T04:58:14.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:14.400 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T04:58:14.657 DEBUG:teuthology.parallel:result is None 2024-01-11T04:58:14.657 INFO:teuthology.run_tasks:Running task install... 2024-01-11T04:58:14.666 DEBUG:teuthology.task.install:project ceph 2024-01-11T04:58:14.666 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'}} 2024-01-11T04:58:14.666 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'} 2024-01-11T04:58:14.666 INFO:teuthology.task.install:Using flavor: default 2024-01-11T04:58:14.673 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-11T04:58:14.674 INFO:teuthology.task.install:extra packages: [] 2024-01-11T04:58:14.674 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'tag': None, 'wait_for_package': False} 2024-01-11T04:58:14.674 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T04:58:14.674 INFO:teuthology.packaging:ref: None 2024-01-11T04:58:14.674 INFO:teuthology.packaging:tag: None 2024-01-11T04:58:14.674 INFO:teuthology.packaging:branch: pacific 2024-01-11T04:58:14.674 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T04:58:14.675 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-01-11T04:58:14.676 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680', 'tag': None, 'wait_for_package': False} 2024-01-11T04:58:14.676 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T04:58:14.676 INFO:teuthology.packaging:ref: None 2024-01-11T04:58:14.676 INFO:teuthology.packaging:tag: None 2024-01-11T04:58:14.677 INFO:teuthology.packaging:branch: pacific 2024-01-11T04:58:14.677 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T04:58:14.677 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-01-11T04:58:14.893 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/ 2024-01-11T04:58:14.894 INFO:teuthology.task.install.rpm:Package version is 16.2.14-417.gcda00bbf 2024-01-11T04:58:14.997 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/ 2024-01-11T04:58:14.998 INFO:teuthology.task.install.rpm:Package version is 16.2.14-417.gcda00bbf 2024-01-11T04:58:15.153 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-11T04:58:15.153 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T04:58:15.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-11T04:58:15.186 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-11T04:58:15.187 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T04:58:15.187 INFO:teuthology.packaging:ref: None 2024-01-11T04:58:15.187 INFO:teuthology.packaging:tag: None 2024-01-11T04:58:15.187 INFO:teuthology.packaging:branch: pacific 2024-01-11T04:58:15.187 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T04:58:15.187 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-11T04:58:15.241 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/cda00bbf2db69ffadb20c123210e286dda76e98e/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-11T04:58:15.241 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T04:58:15.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-11T04:58:15.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-11T04:58:15.274 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-11T04:58:15.274 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T04:58:15.274 INFO:teuthology.packaging:ref: None 2024-01-11T04:58:15.274 INFO:teuthology.packaging:tag: None 2024-01-11T04:58:15.274 INFO:teuthology.packaging:branch: pacific 2024-01-11T04:58:15.275 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T04:58:15.275 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-11T04:58:15.338 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-11T04:58:15.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-11T04:58:15.406 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-01-11T04:58:15.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-01-11T04:58:15.427 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-11T04:58:15.496 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-01-11T04:58:15.497 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-01-11T04:58:15.685 INFO:teuthology.orchestra.run.smithi067.stdout:76 files removed 2024-01-11T04:58:15.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-01-11T04:58:15.768 INFO:teuthology.orchestra.run.smithi169.stdout:76 files removed 2024-01-11T04:58:15.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-01-11T04:58:16.867 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 107 kB/s | 76 kB 00:00 2024-01-11T04:58:16.901 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 109 kB/s | 76 kB 00:00 2024-01-11T04:58:17.192 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 54 kB/s | 15 kB 00:00 2024-01-11T04:58:17.243 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 50 kB/s | 15 kB 00:00 2024-01-11T04:58:17.544 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 5.3 kB/s | 1.4 kB 00:00 2024-01-11T04:58:17.669 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 3.2 kB/s | 1.4 kB 00:00 2024-01-11T04:58:19.794 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - AppStream 16 MB/s | 35 MB 00:02 2024-01-11T04:58:20.236 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - AppStream 14 MB/s | 35 MB 00:02 2024-01-11T04:58:29.179 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 58 MB 00:03 2024-01-11T04:58:30.403 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 58 MB 00:03 2024-01-11T04:58:37.704 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras 60 kB/s | 18 kB 00:00 2024-01-11T04:58:38.087 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 6.9 kB 00:00 2024-01-11T04:58:39.728 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2024-01-11T04:58:40.162 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - Extras common packages 29 kB/s | 6.9 kB 00:00 2024-01-11T04:58:40.547 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 8 - PowerTools 2.9 MB/s | 6.7 MB 00:02 2024-01-11T04:58:42.032 INFO:teuthology.orchestra.run.smithi169.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-01-11T04:58:42.713 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 8 - PowerTools 2.8 MB/s | 6.7 MB 00:02 2024-01-11T04:58:43.433 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-01-11T04:58:44.248 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-11T04:58:46.235 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 16 MB 00:01 2024-01-11T04:58:47.308 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 377 kB/s | 24 kB 00:00 2024-01-11T04:58:50.533 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 393 kB/s | 24 kB 00:00 2024-01-11T04:58:51.983 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T04:58:51.985 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 12 M 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 3.7 M 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 4.2 M 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 6.1 M 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 24 M 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 26 k 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-11T04:58:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 825 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 494 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 3.8 M 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 45 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 99 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 215 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 389 k 2024-01-11T04:58:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 369 k 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 114 k 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout:Install 21 Packages 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 58 M 2024-01-11T04:58:51.988 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T04:58:52.885 INFO:teuthology.orchestra.run.smithi169.stdout:(1/23): ceph-base-16.2.14-417.gcda00bbf.el8.x86 9.0 MB/s | 6.1 MB 00:00 2024-01-11T04:58:52.943 INFO:teuthology.orchestra.run.smithi169.stdout:(2/23): ceph-selinux-16.2.14-417.gcda00bbf.el8. 439 kB/s | 26 kB 00:00 2024-01-11T04:58:53.069 INFO:teuthology.orchestra.run.smithi169.stdout:(3/23): libcephfs2-16.2.14-417.gcda00bbf.el8.x8 6.5 MB/s | 825 kB 00:00 2024-01-11T04:58:53.160 INFO:teuthology.orchestra.run.smithi169.stdout:(4/23): libradosstriper1-16.2.14-417.gcda00bbf. 5.3 MB/s | 494 kB 00:00 2024-01-11T04:58:53.561 INFO:teuthology.orchestra.run.smithi169.stdout:(5/23): librgw2-16.2.14-417.gcda00bbf.el8.x86_6 9.5 MB/s | 3.8 MB 00:00 2024-01-11T04:58:53.611 INFO:teuthology.orchestra.run.smithi169.stdout:(6/23): python3-ceph-argparse-16.2.14-417.gcda0 903 kB/s | 45 kB 00:00 2024-01-11T04:58:53.678 INFO:teuthology.orchestra.run.smithi169.stdout:(7/23): python3-ceph-common-16.2.14-417.gcda00b 1.4 MB/s | 99 kB 00:00 2024-01-11T04:58:53.753 INFO:teuthology.orchestra.run.smithi169.stdout:(8/23): python3-cephfs-16.2.14-417.gcda00bbf.el 2.8 MB/s | 215 kB 00:00 2024-01-11T04:58:53.924 INFO:teuthology.orchestra.run.smithi169.stdout:(9/23): ceph-radosgw-16.2.14-417.gcda00bbf.el8. 6.8 MB/s | 12 MB 00:01 2024-01-11T04:58:53.945 INFO:teuthology.orchestra.run.smithi169.stdout:(10/23): python3-rados-16.2.14-417.gcda00bbf.el 2.0 MB/s | 389 kB 00:00 2024-01-11T04:58:54.004 INFO:teuthology.orchestra.run.smithi169.stdout:(11/23): python3-rbd-16.2.14-417.gcda00bbf.el8. 4.3 MB/s | 369 kB 00:00 2024-01-11T04:58:54.030 INFO:teuthology.orchestra.run.smithi169.stdout:(12/23): python3-rgw-16.2.14-417.gcda00bbf.el8. 1.3 MB/s | 114 kB 00:00 2024-01-11T04:58:54.305 INFO:teuthology.orchestra.run.smithi169.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 944 kB/s | 259 kB 00:00 2024-01-11T04:58:54.531 INFO:teuthology.orchestra.run.smithi169.stdout:(14/23): ceph-common-16.2.14-417.gcda00bbf.el8. 10 MB/s | 24 MB 00:02 2024-01-11T04:58:54.555 INFO:teuthology.orchestra.run.smithi169.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 175 kB/s | 44 kB 00:00 2024-01-11T04:58:54.581 INFO:teuthology.orchestra.run.smithi169.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 677 kB 00:00 2024-01-11T04:58:54.723 INFO:teuthology.orchestra.run.smithi169.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2024-01-11T04:58:54.757 INFO:teuthology.orchestra.run.smithi169.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2024-01-11T04:58:54.782 INFO:teuthology.orchestra.run.smithi169.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-01-11T04:58:54.807 INFO:teuthology.orchestra.run.smithi169.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-01-11T04:58:54.891 INFO:teuthology.orchestra.run.smithi169.stdout:(21/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 130 kB/s | 47 kB 00:00 2024-01-11T04:58:55.116 INFO:teuthology.orchestra.run.smithi169.stdout:(22/23): librados2-16.2.14-417.gcda00bbf.el8.x8 11 MB/s | 3.7 MB 00:00 2024-01-11T04:58:55.300 INFO:teuthology.orchestra.run.smithi169.stdout:(23/23): librbd1-16.2.14-417.gcda00bbf.el8.x86_ 8.5 MB/s | 4.2 MB 00:00 2024-01-11T04:58:55.302 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:58:55.302 INFO:teuthology.orchestra.run.smithi169.stdout:Total 17 MB/s | 58 MB 00:03 2024-01-11T04:58:55.344 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T04:58:55.391 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T04:58:55.391 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T04:58:55.466 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 12 M 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 3.7 M 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 4.2 M 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 6.1 M 2024-01-11T04:58:55.469 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 24 M 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 26 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 825 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 494 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 3.8 M 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-11T04:58:55.470 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 45 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 99 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 215 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 389 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 369 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 114 k 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout:Install 21 Packages 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-01-11T04:58:55.471 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:58:55.472 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 58 M 2024-01-11T04:58:55.472 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T04:58:55.732 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T04:58:55.735 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T04:58:56.765 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T04:58:56.834 INFO:teuthology.orchestra.run.smithi067.stdout:(1/23): ceph-base-16.2.14-417.gcda00bbf.el8.x86 7.0 MB/s | 6.1 MB 00:00 2024-01-11T04:58:56.909 INFO:teuthology.orchestra.run.smithi067.stdout:(2/23): ceph-selinux-16.2.14-417.gcda00bbf.el8. 342 kB/s | 26 kB 00:00 2024-01-11T04:58:57.084 INFO:teuthology.orchestra.run.smithi067.stdout:(3/23): libcephfs2-16.2.14-417.gcda00bbf.el8.x8 4.6 MB/s | 825 kB 00:00 2024-01-11T04:58:57.191 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-01-11T04:58:57.293 INFO:teuthology.orchestra.run.smithi067.stdout:(4/23): ceph-radosgw-16.2.14-417.gcda00bbf.el8. 8.8 MB/s | 12 MB 00:01 2024-01-11T04:58:57.327 INFO:teuthology.orchestra.run.smithi067.stdout:(5/23): libradosstriper1-16.2.14-417.gcda00bbf. 2.0 MB/s | 494 kB 00:00 2024-01-11T04:58:57.399 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-11T04:58:57.402 INFO:teuthology.orchestra.run.smithi067.stdout:(6/23): python3-ceph-argparse-16.2.14-417.gcda0 601 kB/s | 45 kB 00:00 2024-01-11T04:58:57.519 INFO:teuthology.orchestra.run.smithi067.stdout:(7/23): python3-ceph-common-16.2.14-417.gcda00b 847 kB/s | 99 kB 00:00 2024-01-11T04:58:57.619 INFO:teuthology.orchestra.run.smithi067.stdout:(8/23): python3-cephfs-16.2.14-417.gcda00bbf.el 2.1 MB/s | 215 kB 00:00 2024-01-11T04:58:57.678 INFO:teuthology.orchestra.run.smithi067.stdout:(9/23): librgw2-16.2.14-417.gcda00bbf.el8.x86_6 9.9 MB/s | 3.8 MB 00:00 2024-01-11T04:58:57.711 INFO:teuthology.orchestra.run.smithi067.stdout:(10/23): python3-rados-16.2.14-417.gcda00bbf.el 4.1 MB/s | 389 kB 00:00 2024-01-11T04:58:57.745 INFO:teuthology.orchestra.run.smithi067.stdout:(11/23): python3-rbd-16.2.14-417.gcda00bbf.el8. 5.4 MB/s | 369 kB 00:00 2024-01-11T04:58:57.770 INFO:teuthology.orchestra.run.smithi067.stdout:(12/23): python3-rgw-16.2.14-417.gcda00bbf.el8. 1.9 MB/s | 114 kB 00:00 2024-01-11T04:58:57.784 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-11T04:58:57.965 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 3/25 2024-01-11T04:58:58.054 INFO:teuthology.orchestra.run.smithi067.stdout:(13/23): ceph-common-16.2.14-417.gcda00bbf.el8. 12 MB/s | 24 MB 00:02 2024-01-11T04:58:58.058 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 3/25 2024-01-11T04:58:58.179 INFO:teuthology.orchestra.run.smithi067.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.5 MB/s | 677 kB 00:00 2024-01-11T04:58:58.247 INFO:teuthology.orchestra.run.smithi067.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 229 kB/s | 44 kB 00:00 2024-01-11T04:58:58.314 INFO:teuthology.orchestra.run.smithi067.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 478 kB/s | 259 kB 00:00 2024-01-11T04:58:58.314 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 4/25 2024-01-11T04:58:58.489 INFO:teuthology.orchestra.run.smithi067.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-01-11T04:58:58.514 INFO:teuthology.orchestra.run.smithi067.stdout:(18/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 140 kB/s | 47 kB 00:00 2024-01-11T04:58:58.515 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 5/25 2024-01-11T04:58:58.539 INFO:teuthology.orchestra.run.smithi067.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 802 kB/s | 181 kB 00:00 2024-01-11T04:58:58.565 INFO:teuthology.orchestra.run.smithi067.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 781 kB/s | 59 kB 00:00 2024-01-11T04:58:58.723 INFO:teuthology.orchestra.run.smithi067.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 359 kB/s | 75 kB 00:00 2024-01-11T04:58:58.907 INFO:teuthology.orchestra.run.smithi067.stdout:(22/23): librbd1-16.2.14-417.gcda00bbf.el8.x86_ 12 MB/s | 4.2 MB 00:00 2024-01-11T04:58:58.908 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 5/25 2024-01-11T04:58:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 6/25 2024-01-11T04:58:59.076 INFO:teuthology.orchestra.run.smithi067.stdout:(23/23): librados2-16.2.14-417.gcda00bbf.el8.x8 7.0 MB/s | 3.7 MB 00:00 2024-01-11T04:58:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:58:59.079 INFO:teuthology.orchestra.run.smithi067.stdout:Total 16 MB/s | 58 MB 00:03 2024-01-11T04:58:59.124 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T04:58:59.155 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 6/25 2024-01-11T04:58:59.175 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T04:58:59.175 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T04:58:59.350 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2024-01-11T04:58:59.542 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T04:58:59.545 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T04:58:59.812 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-01-11T04:58:59.957 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 9/25 2024-01-11T04:59:00.064 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 9/25 2024-01-11T04:59:00.245 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-01-11T04:59:00.395 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:16.2.14-417.gcda00bbf.el8. 11/25 2024-01-11T04:59:00.555 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T04:59:00.602 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 12/25 2024-01-11T04:59:00.773 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-01-11T04:59:00.810 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 13/25 2024-01-11T04:59:00.942 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-11T04:59:01.057 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 14/25 2024-01-11T04:59:01.198 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_6 15/25 2024-01-11T04:59:01.260 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_6 15/25 2024-01-11T04:59:01.352 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-11T04:59:01.483 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-01-11T04:59:01.525 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 3/25 2024-01-11T04:59:01.608 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 3/25 2024-01-11T04:59:01.656 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-01-11T04:59:01.843 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-01-11T04:59:01.844 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 4/25 2024-01-11T04:59:02.032 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:16.2.14-417.gcda00bbf.el8.x8 19/25 2024-01-11T04:59:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 5/25 2024-01-11T04:59:02.501 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 5/25 2024-01-11T04:59:02.683 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 6/25 2024-01-11T04:59:02.754 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 6/25 2024-01-11T04:59:02.930 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2024-01-11T04:59:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-01-11T04:59:03.592 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 9/25 2024-01-11T04:59:03.669 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 9/25 2024-01-11T04:59:03.856 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-01-11T04:59:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:16.2.14-417.gcda00bbf.el8. 11/25 2024-01-11T04:59:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 12/25 2024-01-11T04:59:04.411 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 13/25 2024-01-11T04:59:04.516 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 20/25 2024-01-11T04:59:04.612 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 14/25 2024-01-11T04:59:04.756 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 20/25 2024-01-11T04:59:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_6 15/25 2024-01-11T04:59:04.829 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_6 15/25 2024-01-11T04:59:05.062 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-01-11T04:59:05.234 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-01-11T04:59:05.409 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-01-11T04:59:05.464 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 20/25 2024-01-11T04:59:05.600 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:16.2.14-417.gcda00bbf.el8.x8 19/25 2024-01-11T04:59:05.773 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 21/25 2024-01-11T04:59:05.910 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 21/25 2024-01-11T04:59:06.047 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 22/25 2024-01-11T04:59:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 20/25 2024-01-11T04:59:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 20/25 2024-01-11T04:59:09.218 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 20/25 2024-01-11T04:59:09.542 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 21/25 2024-01-11T04:59:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 21/25 2024-01-11T04:59:09.983 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 22/25 2024-01-11T04:59:25.048 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 22/25 2024-01-11T04:59:25.048 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-01-11T04:59:25.048 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /run 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /dev 2024-01-11T04:59:25.049 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:26.236 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 23/25 2024-01-11T04:59:26.315 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 23/25 2024-01-11T04:59:26.315 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-11T04:59:26.546 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-11T04:59:26.546 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 1/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 2/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 3/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 4/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 5/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_6 6/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 7/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:16.2.14-417.gcda00bbf.el8. 8/25 2024-01-11T04:59:27.182 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:16.2.14-417.gcda00bbf.el8.x8 9/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 10/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 11/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 12/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 13/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-01-11T04:59:27.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 22/25 2024-01-11T04:59:27.184 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-01-11T04:59:27.184 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 24/25 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-11T04:59:28.479 INFO:teuthology.orchestra.run.smithi169.stdout: leveldb-1.22-1.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-11T04:59:28.480 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:28.481 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:28.481 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T04:59:28.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-01-11T04:59:29.583 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:42 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 22/25 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /run 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-01-11T04:59:30.048 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-01-11T04:59:30.049 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2024-01-11T04:59:30.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:31.300 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T04:59:31.301 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:59:31.301 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T04:59:31.301 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 52 M 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 174 k 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-01-11T04:59:31.302 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:31.303 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 53 M 2024-01-11T04:59:31.303 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 196 M 2024-01-11T04:59:31.303 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T04:59:31.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 23/25 2024-01-11T04:59:31.423 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 23/25 2024-01-11T04:59:31.423 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-11T04:59:31.591 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-11T04:59:31.591 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-11T04:59:31.689 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): libcephsqlite-16.2.14-417.gcda00bbf.el8. 932 kB/s | 174 kB 00:00 2024-01-11T04:59:31.756 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 802 kB/s | 203 kB 00:00 2024-01-11T04:59:31.882 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-01-11T04:59:31.940 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 747 kB/s | 187 kB 00:00 2024-01-11T04:59:31.965 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 845 kB/s | 71 kB 00:00 2024-01-11T04:59:32.291 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 1/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 2/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 3/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 4/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 5/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_6 6/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 7/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:16.2.14-417.gcda00bbf.el8. 8/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:16.2.14-417.gcda00bbf.el8.x8 9/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 10/25 2024-01-11T04:59:32.292 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 11/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 12/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 13/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 22/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-01-11T04:59:32.293 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 24/25 2024-01-11T04:59:33.698 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-01-11T04:59:33.698 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:33.698 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-01-11T04:59:33.698 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.698 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: leveldb-1.22-1.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-11T04:59:33.699 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:33.700 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T04:59:33.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-01-11T04:59:34.349 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-test-16.2.14-417.gcda00bbf.el8.x86_ 18 MB/s | 52 MB 00:02 2024-01-11T04:59:34.351 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:59:34.352 INFO:teuthology.orchestra.run.smithi169.stdout:Total 17 MB/s | 53 MB 00:03 2024-01-11T04:59:34.395 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T04:59:34.415 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T04:59:34.416 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T04:59:34.705 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T04:59:34.708 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T04:59:34.781 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:44 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T04:59:35.175 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T04:59:35.397 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-11T04:59:35.570 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-11T04:59:35.709 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-11T04:59:35.771 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-11T04:59:35.965 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-11T04:59:36.159 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 5/6 2024-01-11T04:59:36.604 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T04:59:36.605 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:59:36.605 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T04:59:36.605 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 52 M 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 174 k 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-01-11T04:59:36.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:36.607 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 53 M 2024-01-11T04:59:36.607 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 196 M 2024-01-11T04:59:36.607 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T04:59:36.854 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): libcephsqlite-16.2.14-417.gcda00bbf.el8. 845 kB/s | 174 kB 00:00 2024-01-11T04:59:36.913 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 769 kB/s | 203 kB 00:00 2024-01-11T04:59:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2024-01-11T04:59:37.088 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-01-11T04:59:37.105 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 748 kB/s | 187 kB 00:00 2024-01-11T04:59:39.139 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): ceph-test-16.2.14-417.gcda00bbf.el8.x86_ 21 MB/s | 52 MB 00:02 2024-01-11T04:59:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:59:39.141 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 53 MB 00:02 2024-01-11T04:59:39.187 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T04:59:39.209 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T04:59:39.209 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T04:59:39.511 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T04:59:39.514 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T04:59:40.024 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T04:59:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-11T04:59:40.558 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-11T04:59:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-11T04:59:40.753 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-11T04:59:40.774 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 5/6 2024-01-11T04:59:40.920 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-11T04:59:41.032 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 6/6 2024-01-11T04:59:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 5/6 2024-01-11T04:59:41.316 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 6/6 2024-01-11T04:59:41.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 1/6 2024-01-11T04:59:41.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 2/6 2024-01-11T04:59:41.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-11T04:59:41.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-11T04:59:41.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-8.el8.x86_64 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:42.080 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T04:59:42.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-01-11T04:59:43.184 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:56 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T04:59:44.908 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 6.4 k 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T04:59:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 2.4 M 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 1.6 M 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 245 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 5.0 M 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 18 M 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-11T04:59:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-11T04:59:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout:Install 32 Packages 2024-01-11T04:59:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T04:59:44.916 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 M 2024-01-11T04:59:44.916 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 101 M 2024-01-11T04:59:44.916 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T04:59:45.308 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-16.2.14-417.gcda00bbf.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2024-01-11T04:59:45.450 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-mgr-16.2.14-417.gcda00bbf.el8.x86_ 6.0 MB/s | 1.6 MB 00:00 2024-01-11T04:59:45.961 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 5/6 2024-01-11T04:59:46.184 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): ceph-mds-16.2.14-417.gcda00bbf.el8.x86_ 2.4 MB/s | 2.4 MB 00:01 2024-01-11T04:59:46.186 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 6/6 2024-01-11T04:59:46.293 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): ceph-mgr-modules-core-16.2.14-417.gcda0 2.2 MB/s | 245 kB 00:00 2024-01-11T04:59:46.360 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): ceph-mon-16.2.14-417.gcda00bbf.el8.x86_ 4.8 MB/s | 5.0 MB 00:01 2024-01-11T04:59:46.483 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 6/6 2024-01-11T04:59:46.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 1/6 2024-01-11T04:59:46.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 2/6 2024-01-11T04:59:46.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-11T04:59:46.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-11T04:59:46.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-11T04:59:46.543 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): ceph-osd-16.2.14-417.gcda00bbf.el8.x86_ 17 MB/s | 18 MB 00:01 2024-01-11T04:59:46.585 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 538 kB/s | 157 kB 00:00 2024-01-11T04:59:46.652 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 855 kB/s | 57 kB 00:00 2024-01-11T04:59:46.677 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 122 kB/s | 39 kB 00:00 2024-01-11T04:59:46.761 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.1 MB/s | 457 kB 00:00 2024-01-11T04:59:46.786 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 426 kB/s | 103 kB 00:00 2024-01-11T04:59:47.220 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 459 kB/s | 249 kB 00:00 2024-01-11T04:59:47.245 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 238 kB/s | 109 kB 00:00 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-8.el8.x86_64 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-11T04:59:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-11T04:59:47.302 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:47.302 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T04:59:47.387 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2024-01-11T04:59:47.421 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 972 kB/s | 641 kB 00:00 2024-01-11T04:59:47.446 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 923 kB/s | 185 kB 00:00 2024-01-11T04:59:47.496 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2024-01-11T04:59:47.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-01-11T04:59:47.522 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2024-01-11T04:59:47.547 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2024-01-11T04:59:47.572 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 965 kB/s | 49 kB 00:00 2024-01-11T04:59:47.598 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-01-11T04:59:47.640 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-01-11T04:59:47.665 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-01-11T04:59:47.690 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.3 MB/s | 283 kB 00:00 2024-01-11T04:59:47.744 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 180 kB/s | 19 kB 00:00 2024-01-11T04:59:47.774 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-01-11T04:59:47.799 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2024-01-11T04:59:47.824 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 345 kB/s | 27 kB 00:00 2024-01-11T04:59:47.867 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2024-01-11T04:59:47.892 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-01-11T04:59:47.917 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 927 kB/s | 85 kB 00:00 2024-01-11T04:59:47.942 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2024-01-11T04:59:47.943 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:59:47.944 INFO:teuthology.orchestra.run.smithi169.stdout:Total 10 MB/s | 31 MB 00:03 2024-01-11T04:59:48.028 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T04:59:48.067 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T04:59:48.067 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T04:59:48.382 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T04:59:48.384 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T04:59:48.407 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:58 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T04:59:49.526 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T04:59:49.753 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-11T04:59:49.977 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-11T04:59:50.235 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T04:59:50.238 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:59:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T04:59:50.238 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:59:50.238 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 6.4 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 2.4 M 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 1.6 M 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 245 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 5.0 M 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 18 M 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-11T04:59:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-11T04:59:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout:Install 32 Packages 2024-01-11T04:59:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T04:59:50.242 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 31 M 2024-01-11T04:59:50.242 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 101 M 2024-01-11T04:59:50.242 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T04:59:50.248 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-11T04:59:50.488 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-11T04:59:50.678 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-11T04:59:50.876 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-11T04:59:51.093 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-11T04:59:51.263 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-11T04:59:51.488 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-11T04:59:51.726 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-11T04:59:51.882 INFO:teuthology.orchestra.run.smithi067.stdout:(1/32): ceph-16.2.14-417.gcda00bbf.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2024-01-11T04:59:52.006 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-11T04:59:52.041 INFO:teuthology.orchestra.run.smithi067.stdout:(2/32): ceph-mds-16.2.14-417.gcda00bbf.el8.x86_ 7.3 MB/s | 2.4 MB 00:00 2024-01-11T04:59:52.257 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-11T04:59:52.600 INFO:teuthology.orchestra.run.smithi067.stdout:(3/32): ceph-mgr-16.2.14-417.gcda00bbf.el8.x86_ 1.8 MB/s | 1.6 MB 00:00 2024-01-11T04:59:52.626 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-11T04:59:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-11T04:59:53.000 INFO:teuthology.orchestra.run.smithi067.stdout:(4/32): ceph-osd-16.2.14-417.gcda00bbf.el8.x86_ 19 MB/s | 18 MB 00:00 2024-01-11T04:59:53.025 INFO:teuthology.orchestra.run.smithi067.stdout:(5/32): ceph-mgr-modules-core-16.2.14-417.gcda0 575 kB/s | 245 kB 00:00 2024-01-11T04:59:53.081 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-11T04:59:53.167 INFO:teuthology.orchestra.run.smithi067.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 274 kB/s | 39 kB 00:00 2024-01-11T04:59:53.251 INFO:teuthology.orchestra.run.smithi067.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 627 kB/s | 157 kB 00:00 2024-01-11T04:59:53.268 INFO:teuthology.orchestra.run.smithi067.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2024-01-11T04:59:53.360 INFO:teuthology.orchestra.run.smithi067.stdout:(9/32): ceph-mon-16.2.14-417.gcda00bbf.el8.x86_ 3.4 MB/s | 5.0 MB 00:01 2024-01-11T04:59:53.385 INFO:teuthology.orchestra.run.smithi067.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 424 kB/s | 57 kB 00:00 2024-01-11T04:59:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-11T04:59:53.469 INFO:teuthology.orchestra.run.smithi067.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2024-01-11T04:59:53.666 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-11T04:59:53.903 INFO:teuthology.orchestra.run.smithi067.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 252 kB/s | 109 kB 00:00 2024-01-11T04:59:53.919 INFO:teuthology.orchestra.run.smithi067.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 445 kB/s | 249 kB 00:00 2024-01-11T04:59:53.995 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-11T04:59:54.053 INFO:teuthology.orchestra.run.smithi067.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 961 kB/s | 641 kB 00:00 2024-01-11T04:59:54.203 INFO:teuthology.orchestra.run.smithi067.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 145 kB/s | 44 kB 00:00 2024-01-11T04:59:54.228 INFO:teuthology.orchestra.run.smithi067.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 600 kB/s | 185 kB 00:00 2024-01-11T04:59:54.287 INFO:teuthology.orchestra.run.smithi067.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2024-01-11T04:59:54.288 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-11T04:59:54.346 INFO:teuthology.orchestra.run.smithi067.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 302 kB/s | 18 kB 00:00 2024-01-11T04:59:54.379 INFO:teuthology.orchestra.run.smithi067.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2024-01-11T04:59:54.405 INFO:teuthology.orchestra.run.smithi067.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 491 kB/s | 173 kB 00:00 2024-01-11T04:59:54.430 INFO:teuthology.orchestra.run.smithi067.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 580 kB/s | 49 kB 00:00 2024-01-11T04:59:54.455 INFO:teuthology.orchestra.run.smithi067.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 779 kB/s | 59 kB 00:00 2024-01-11T04:59:54.476 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-11T04:59:54.489 INFO:teuthology.orchestra.run.smithi067.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-01-11T04:59:54.514 INFO:teuthology.orchestra.run.smithi067.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 320 kB/s | 19 kB 00:00 2024-01-11T04:59:54.539 INFO:teuthology.orchestra.run.smithi067.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-01-11T04:59:54.565 INFO:teuthology.orchestra.run.smithi067.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-01-11T04:59:54.581 INFO:teuthology.orchestra.run.smithi067.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 435 kB/s | 29 kB 00:00 2024-01-11T04:59:54.598 INFO:teuthology.orchestra.run.smithi067.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 468 kB/s | 27 kB 00:00 2024-01-11T04:59:54.649 INFO:teuthology.orchestra.run.smithi067.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2024-01-11T04:59:54.674 INFO:teuthology.orchestra.run.smithi067.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-01-11T04:59:54.695 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-11T04:59:54.699 INFO:teuthology.orchestra.run.smithi067.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 847 kB/s | 85 kB 00:00 2024-01-11T04:59:54.716 INFO:teuthology.orchestra.run.smithi067.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 343 kB/s | 23 kB 00:00 2024-01-11T04:59:54.717 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T04:59:54.718 INFO:teuthology.orchestra.run.smithi067.stdout:Total 7.0 MB/s | 31 MB 00:04 2024-01-11T04:59:54.806 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T04:59:54.847 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T04:59:54.847 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T04:59:54.943 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-11T04:59:55.183 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T04:59:55.186 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T04:59:55.193 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-11T04:59:55.444 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-11T04:59:55.760 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-11T04:59:56.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-11T04:59:56.309 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T04:59:56.502 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-417.gcda00bbf.el8. 27/32 2024-01-11T04:59:56.587 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-11T04:59:56.697 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 28/32 2024-01-11T04:59:56.810 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-11T04:59:57.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-11T04:59:57.259 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-11T04:59:57.456 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-11T04:59:57.629 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-11T04:59:57.797 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-11T04:59:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-11T04:59:58.204 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 28/32 2024-01-11T04:59:58.210 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-11T04:59:58.389 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 29/32 2024-01-11T04:59:58.439 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-11T04:59:58.643 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-11T04:59:58.818 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 29/32 2024-01-11T04:59:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-11T04:59:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 30/32 2024-01-11T04:59:59.207 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 30/32 2024-01-11T04:59:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-11T04:59:59.406 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 31/32 2024-01-11T04:59:59.423 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 31/32 2024-01-11T04:59:59.506 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-11T04:59:59.605 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 32/32 2024-01-11T04:59:59.702 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-11T04:59:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 32/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 1/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 2/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 3/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 4/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 5/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-417.gcda00bbf.el8. 6/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-11T04:59:59.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-11T04:59:59.999 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-11T05:00:00.000 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-11T05:00:00.000 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-11T05:00:00.000 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-11T05:00:00.000 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-11T05:00:00.001 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-11T05:00:00.001 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-11T05:00:00.170 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-11T05:00:00.467 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-11T05:00:00.675 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-11T05:00:00.830 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-11T05:00:00.990 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-11T05:00:01.207 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-11T05:00:01.419 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-11T05:00:01.637 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:01.754 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-11T05:00:01.755 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-2.6-1.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-11T05:00:01.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-11T05:00:01.757 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-11T05:00:01.757 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:01.757 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:01.936 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-11T05:00:01.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-01-11T05:00:02.263 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-11T05:00:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-417.gcda00bbf.el8. 27/32 2024-01-11T05:00:02.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 28/32 2024-01-11T05:00:02.847 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:15 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:04.330 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:00:04.425 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 28/32 2024-01-11T05:00:04.568 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:04.569 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:00:04.569 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:04.593 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 29/32 2024-01-11T05:00:04.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-01-11T05:00:05.046 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 29/32 2024-01-11T05:00:05.552 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:18 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:05.827 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 30/32 2024-01-11T05:00:06.062 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 30/32 2024-01-11T05:00:06.251 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 31/32 2024-01-11T05:00:06.270 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 31/32 2024-01-11T05:00:06.393 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 32/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 32/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 1/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 2/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 3/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 4/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 5/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-417.gcda00bbf.el8. 6/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-11T05:00:06.820 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-11T05:00:06.821 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-11T05:00:06.822 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-11T05:00:06.822 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-11T05:00:06.822 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-11T05:00:06.822 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-11T05:00:06.822 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-11T05:00:07.282 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 85 k 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:00:07.283 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:07.284 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:00:07.284 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:07.284 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-01-11T05:00:07.284 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 352 k 2024-01-11T05:00:07.284 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:00:07.466 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-16.2.14-417.gcda00bbf.el8.noarch.rpm 470 kB/s | 85 kB 00:00 2024-01-11T05:00:07.467 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:07.467 INFO:teuthology.orchestra.run.smithi169.stdout:Total 466 kB/s | 85 kB 00:00 2024-01-11T05:00:07.467 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:00:07.474 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:00:07.474 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:00:07.476 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:00:07.479 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:00:07.524 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:08.012 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-2.6-1.el8.noarch 2024-01-11T05:00:08.013 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:08.014 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:08.024 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:08.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-01-11T05:00:08.225 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:08.454 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:09.056 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:19 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:09.137 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:09.137 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:09.137 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:09.137 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:09.137 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:09.137 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:09.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-01-11T05:00:10.200 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:23 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:10.643 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:00:10.897 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:10.898 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:00:10.898 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:10.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-01-11T05:00:11.888 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:11.889 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:21 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:11.889 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 161 k 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 161 k 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 470 k 2024-01-11T05:00:11.890 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:00:12.470 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-16.2.14-417.gcda00b 278 kB/s | 161 kB 00:00 2024-01-11T05:00:12.471 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:12.471 INFO:teuthology.orchestra.run.smithi169.stdout:Total 277 kB/s | 161 kB 00:00 2024-01-11T05:00:12.471 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:00:12.482 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:00:12.482 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:00:12.530 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:00:12.533 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:00:12.655 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:00:12.854 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf. 1/1 2024-01-11T05:00:13.298 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf. 1/1 2024-01-11T05:00:13.723 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 85 k 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:00:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:13.725 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 85 k 2024-01-11T05:00:13.725 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 352 k 2024-01-11T05:00:13.725 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:00:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-16.2.14-417.gcda00bbf.el8.noarch.rpm 330 kB/s | 85 kB 00:00 2024-01-11T05:00:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:Total 328 kB/s | 85 kB 00:00 2024-01-11T05:00:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:00:13.992 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:00:13.992 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:00:13.994 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:00:13.998 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:00:14.015 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf. 1/1 2024-01-11T05:00:14.016 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:14.016 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:14.016 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:14.016 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:14.016 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:00:14.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-01-11T05:00:14.544 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:14.705 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:14.943 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:15.104 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:28 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/1 2024-01-11T05:00:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:15.655 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:00:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:15.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:15.655 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:15.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-01-11T05:00:16.603 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:00:16.801 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:26 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:16.842 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:16.843 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:00:16.843 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:16.943 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-01-11T05:00:17.790 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:30 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:18.640 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:18.640 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:18.640 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-01-11T05:00:18.640 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:18.640 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:00:18.640 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 161 k 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 161 k 2024-01-11T05:00:18.641 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 470 k 2024-01-11T05:00:18.642 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:00:18.938 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-16.2.14-417.gcda00b 544 kB/s | 161 kB 00:00 2024-01-11T05:00:18.939 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:18.939 INFO:teuthology.orchestra.run.smithi067.stdout:Total 542 kB/s | 161 kB 00:00 2024-01-11T05:00:18.939 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:00:18.950 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:00:18.951 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:00:19.002 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:00:19.005 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:00:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:00:19.283 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf. 1/1 2024-01-11T05:00:19.527 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:19.528 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:19.528 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T05:00:19.528 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:19.528 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:00:19.528 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 3.1 M 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 24 k 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 15 k 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-01-11T05:00:19.529 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:19.530 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.4 M 2024-01-11T05:00:19.530 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 22 M 2024-01-11T05:00:19.530 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:00:19.715 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf. 1/1 2024-01-11T05:00:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf. 1/1 2024-01-11T05:00:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:20.463 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:00:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:00:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:20.463 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:20.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-01-11T05:00:21.289 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): ceph-prometheus-alerts-16.2.14-417.gcda0 94 kB/s | 15 kB 00:00 2024-01-11T05:00:21.339 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): ceph-grafana-dashboards-16.2.14-417.gcda 117 kB/s | 24 kB 00:00 2024-01-11T05:00:21.514 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): ceph-mgr-dashboard-16.2.14-417.gcda00bbf 8.1 MB/s | 3.1 MB 00:00 2024-01-11T05:00:21.567 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:31 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:21.673 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 101 kB/s | 34 kB 00:00 2024-01-11T05:00:21.698 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 104 kB/s | 43 kB 00:00 2024-01-11T05:00:21.965 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 437 kB/s | 196 kB 00:00 2024-01-11T05:00:21.968 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:21.969 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.4 MB/s | 3.4 MB 00:02 2024-01-11T05:00:21.981 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:00:21.992 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:00:21.992 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:00:22.174 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:00:22.177 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:00:22.467 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:00:22.742 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-11T05:00:23.045 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-11T05:00:23.151 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:00:23.296 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-11T05:00:23.404 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:00:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:23.425 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-417.gcda00bbf.el8.n 4/6 2024-01-11T05:00:23.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-01-11T05:00:24.380 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:34 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:24.468 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-417.gcda00bbf.el8. 5/6 2024-01-11T05:00:24.743 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarc 6/6 2024-01-11T05:00:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarc 6/6 2024-01-11T05:00:25.019 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-417.gcda00bbf.el8. 1/6 2024-01-11T05:00:25.020 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarc 2/6 2024-01-11T05:00:25.020 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-417.gcda00bbf.el8.n 3/6 2024-01-11T05:00:25.020 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-11T05:00:25.020 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-11T05:00:25.769 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-11T05:00:25.770 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-11T05:00:25.770 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:25.770 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:25.898 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-11T05:00:26.218 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 3.1 M 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 24 k 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 15 k 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-11T05:00:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.4 M 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 22 M 2024-01-11T05:00:26.221 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:00:26.753 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:39 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:27.140 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-grafana-dashboards-16.2.14-417.gcda 169 kB/s | 24 kB 00:00 2024-01-11T05:00:27.240 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): ceph-prometheus-alerts-16.2.14-417.gcda0 61 kB/s | 15 kB 00:00 2024-01-11T05:00:27.474 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 144 kB/s | 34 kB 00:00 2024-01-11T05:00:27.524 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): ceph-mgr-dashboard-16.2.14-417.gcda00bbf 5.9 MB/s | 3.1 MB 00:00 2024-01-11T05:00:27.541 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 106 kB/s | 43 kB 00:00 2024-01-11T05:00:27.716 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 811 kB/s | 196 kB 00:00 2024-01-11T05:00:27.718 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:27.718 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.3 MB/s | 3.4 MB 00:01 2024-01-11T05:00:27.731 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:00:27.743 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:00:27.744 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:00:27.920 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:00:27.923 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:00:28.288 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:00:28.470 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 8.6 M 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-11T05:00:28.471 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout:Install 8 Packages 2024-01-11T05:00:28.472 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:28.473 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-01-11T05:00:28.473 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 210 M 2024-01-11T05:00:28.473 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:00:28.474 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-11T05:00:28.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-11T05:00:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-11T05:00:29.010 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-417.gcda00bbf.el8.n 4/6 2024-01-11T05:00:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-417.gcda00bbf.el8. 5/6 2024-01-11T05:00:30.332 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarc 6/6 2024-01-11T05:00:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarc 6/6 2024-01-11T05:00:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-417.gcda00bbf.el8. 1/6 2024-01-11T05:00:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarc 2/6 2024-01-11T05:00:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-417.gcda00bbf.el8.n 3/6 2024-01-11T05:00:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-11T05:00:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-11T05:00:30.863 INFO:teuthology.orchestra.run.smithi169.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 8.9 MB/s | 4.7 MB 00:00 2024-01-11T05:00:30.947 INFO:teuthology.orchestra.run.smithi169.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.9 MB/s | 4.9 MB 00:00 2024-01-11T05:00:31.056 INFO:teuthology.orchestra.run.smithi169.stdout:(3/8): ceph-mgr-diskprediction-local-16.2.14-41 12 MB/s | 8.6 MB 00:00 2024-01-11T05:00:31.081 INFO:teuthology.orchestra.run.smithi169.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-01-11T05:00:31.156 INFO:teuthology.orchestra.run.smithi169.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2024-01-11T05:00:31.386 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-11T05:00:31.386 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:31.386 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:31.387 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:31.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-11T05:00:31.640 INFO:teuthology.orchestra.run.smithi169.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 357 kB/s | 172 kB 00:00 2024-01-11T05:00:31.723 INFO:teuthology.orchestra.run.smithi169.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2024-01-11T05:00:32.007 INFO:teuthology.orchestra.run.smithi169.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2024-01-11T05:00:32.007 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:32.008 INFO:teuthology.orchestra.run.smithi169.stdout:Total 10 MB/s | 37 MB 00:03 2024-01-11T05:00:32.237 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:00:32.256 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:00:32.256 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:00:32.395 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:42 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:32.582 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:00:32.585 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:00:33.176 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:00:33.375 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-11T05:00:33.496 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-11T05:00:33.667 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-11T05:00:34.246 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:34.247 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-01-11T05:00:34.247 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T05:00:34.247 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-01-11T05:00:34.247 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:00:34.247 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 8.6 M 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout:Install 8 Packages 2024-01-11T05:00:34.248 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:34.249 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 37 M 2024-01-11T05:00:34.249 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 210 M 2024-01-11T05:00:34.249 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:00:34.414 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] openblas-0.3.15-4.el8.x86_64.rpm: Status code: 500 for http://tx-mirror.tier.net/centos/8-stream/AppStream/x86_64/os/Packages/openblas-0.3.15-4.el8.x86_64.rpm (IP: 181.215.47.106) 2024-01-11T05:00:34.515 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-11T05:00:34.719 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-11T05:00:34.873 INFO:teuthology.orchestra.run.smithi067.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.0 MB/s | 4.9 MB 00:00 2024-01-11T05:00:35.106 INFO:teuthology.orchestra.run.smithi067.stdout:(2/8): ceph-mgr-diskprediction-local-16.2.14-41 11 MB/s | 8.6 MB 00:00 2024-01-11T05:00:35.173 INFO:teuthology.orchestra.run.smithi067.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2024-01-11T05:00:35.313 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-11T05:00:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 4.3 MB/s | 4.7 MB 00:01 2024-01-11T05:00:35.450 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-11T05:00:35.465 INFO:teuthology.orchestra.run.smithi067.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 627 kB/s | 225 kB 00:00 2024-01-11T05:00:36.008 INFO:teuthology.orchestra.run.smithi067.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 17 MB/s | 14 MB 00:00 2024-01-11T05:00:36.032 INFO:teuthology.orchestra.run.smithi067.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 304 kB/s | 172 kB 00:00 2024-01-11T05:00:36.174 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-11T05:00:36.225 INFO:teuthology.orchestra.run.smithi067.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 823 kB/s | 645 kB 00:00 2024-01-11T05:00:36.225 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:36.225 INFO:teuthology.orchestra.run.smithi067.stdout:Total 19 MB/s | 37 MB 00:01 2024-01-11T05:00:36.464 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:00:36.483 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:00:36.483 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:00:36.528 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-11T05:00:36.766 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-11T05:00:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:00:36.819 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:00:37.390 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:00:37.531 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-11T05:00:37.670 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-11T05:00:37.797 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-11T05:00:38.435 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-11T05:00:38.518 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-11T05:00:38.589 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-11T05:00:39.238 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-11T05:00:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-11T05:00:39.806 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-11T05:00:39.966 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bb 8/8 2024-01-11T05:00:40.284 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bb 8/8 2024-01-11T05:00:40.284 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bb 1/8 2024-01-11T05:00:40.284 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-11T05:00:40.285 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-11T05:00:40.285 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-11T05:00:40.285 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-11T05:00:40.285 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-11T05:00:40.285 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-11T05:00:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-11T05:00:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-11T05:00:40.847 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-11T05:00:41.019 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:41.020 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:41.153 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-01-11T05:00:42.017 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:55 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:42.731 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-11T05:00:43.746 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:43.747 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 27 k 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-11T05:00:43.748 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-01-11T05:00:43.749 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:43.750 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.9 M 2024-01-11T05:00:43.750 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 24 M 2024-01-11T05:00:43.750 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:00:44.103 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-11T05:00:44.263 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): ceph-mgr-rook-16.2.14-417.gcda00bbf.el8 199 kB/s | 27 kB 00:00 2024-01-11T05:00:44.279 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 172 kB/s | 27 kB 00:00 2024-01-11T05:00:44.280 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bb 8/8 2024-01-11T05:00:44.297 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 116 kB/s | 20 kB 00:00 2024-01-11T05:00:44.405 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 888 kB/s | 126 kB 00:00 2024-01-11T05:00:44.439 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 776 kB/s | 110 kB 00:00 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bb 8/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bb 1/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-11T05:00:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-11T05:00:44.681 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 136 kB/s | 33 kB 00:00 2024-01-11T05:00:44.759 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 121 kB/s | 43 kB 00:00 2024-01-11T05:00:44.790 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 314 kB/s | 155 kB 00:00 2024-01-11T05:00:44.815 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2024-01-11T05:00:44.982 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 385 kB/s | 64 kB 00:00 2024-01-11T05:00:45.066 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 263 kB/s | 81 kB 00:00 2024-01-11T05:00:45.091 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 564 kB/s | 61 kB 00:00 2024-01-11T05:00:45.393 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-11T05:00:45.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-11T05:00:45.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-11T05:00:45.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-11T05:00:45.395 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:45.395 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:45.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-01-11T05:00:45.592 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.5 MB/s | 1.2 MB 00:00 2024-01-11T05:00:45.592 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:45.592 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2024-01-11T05:00:45.651 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:00:45.667 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:00:45.667 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:00:45.754 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:00:45.757 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:00:46.356 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:00:46.416 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:56 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:46.592 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-11T05:00:46.815 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-11T05:00:47.044 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-11T05:00:47.296 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-11T05:00:47.505 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-11T05:00:47.763 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-11T05:00:48.047 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-11T05:00:48.252 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 27 k 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-11T05:00:48.254 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:48.255 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-01-11T05:00:48.256 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:48.256 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.9 M 2024-01-11T05:00:48.256 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 24 M 2024-01-11T05:00:48.256 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:00:48.320 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-11T05:00:48.747 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): ceph-mgr-rook-16.2.14-417.gcda00bbf.el8 192 kB/s | 27 kB 00:00 2024-01-11T05:00:48.772 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 158 kB/s | 27 kB 00:00 2024-01-11T05:00:48.797 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 103 kB/s | 20 kB 00:00 2024-01-11T05:00:48.898 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 839 kB/s | 126 kB 00:00 2024-01-11T05:00:48.939 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 776 kB/s | 110 kB 00:00 2024-01-11T05:00:49.082 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2024-01-11T05:00:49.149 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 245 kB/s | 16 kB 00:00 2024-01-11T05:00:49.224 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2024-01-11T05:00:49.283 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 609 kB/s | 81 kB 00:00 2024-01-11T05:00:49.308 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 304 kB/s | 155 kB 00:00 2024-01-11T05:00:49.358 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 854 kB/s | 64 kB 00:00 2024-01-11T05:00:49.403 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-11T05:00:49.492 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 333 kB/s | 61 kB 00:00 2024-01-11T05:00:49.592 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-01-11T05:00:49.592 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:49.593 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-01-11T05:00:49.655 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:00:49.671 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:00:49.672 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:00:49.679 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-11T05:00:49.766 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:00:49.769 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:00:49.904 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-11T05:00:50.097 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-11T05:00:50.272 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 13/13 2024-01-11T05:00:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:00:50.551 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-11T05:00:50.573 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 13/13 2024-01-11T05:00:50.573 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 1/13 2024-01-11T05:00:50.573 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-11T05:00:50.573 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-11T05:00:50.574 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-11T05:00:50.739 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-11T05:00:50.951 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-11T05:00:51.186 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-11T05:00:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-11T05:00:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:51.352 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:51.352 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:51.353 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:51.394 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-11T05:00:51.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-01-11T05:00:51.594 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-11T05:00:51.845 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-11T05:00:52.093 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-11T05:00:52.346 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:05 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:00:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-11T05:00:53.403 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-11T05:00:53.628 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-11T05:00:53.804 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-11T05:00:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 13/13 2024-01-11T05:00:54.082 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 116 k 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-11T05:00:54.083 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.8 M 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 24 M 2024-01-11T05:00:54.084 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 13/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 1/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-11T05:00:54.284 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-11T05:00:54.285 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-11T05:00:54.285 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-11T05:00:54.285 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-11T05:00:54.285 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-11T05:00:54.399 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): ceph-mgr-cephadm-16.2.14-417.gcda00bbf.e 577 kB/s | 116 kB 00:00 2024-01-11T05:00:54.525 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 170 kB/s | 21 kB 00:00 2024-01-11T05:00:54.550 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2024-01-11T05:00:54.772 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.3 MB/s | 4.8 MB 00:00 2024-01-11T05:00:54.943 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 124 kB/s | 49 kB 00:00 2024-01-11T05:00:55.086 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 633 kB/s | 355 kB 00:00 2024-01-11T05:00:55.087 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:55.087 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.8 MB/s | 5.8 MB 00:01 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-11T05:00:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:55.098 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:00:55.142 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:00:55.152 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:00:55.152 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:00:55.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-01-11T05:00:55.255 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:00:55.258 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:00:55.550 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:00:55.828 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-11T05:00:56.107 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-11T05:00:56.109 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:06 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:00:57.075 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-11T05:00:57.408 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-11T05:00:57.688 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-11T05:00:57.871 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 6/6 2024-01-11T05:00:57.939 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:00:57.940 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:57.940 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T05:00:57.940 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:57.940 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:16.2.14-417.gcda00bbf.el8 ceph-noarch 116 k 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-01-11T05:00:57.941 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:00:57.942 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 5.8 M 2024-01-11T05:00:57.942 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 24 M 2024-01-11T05:00:57.942 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:00:58.160 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 6/6 2024-01-11T05:00:58.160 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/6 2024-01-11T05:00:58.160 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-11T05:00:58.160 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-11T05:00:58.160 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-11T05:00:58.160 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-11T05:00:58.550 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-mgr-cephadm-16.2.14-417.gcda00bbf.e 384 kB/s | 116 kB 00:00 2024-01-11T05:00:58.575 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.6 MB/s | 538 kB 00:00 2024-01-11T05:00:58.792 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 87 kB/s | 21 kB 00:00 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-apipkg-1.5-6.el8.noarch 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-01-11T05:00:58.886 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-11T05:00:58.887 INFO:teuthology.orchestra.run.smithi169.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-01-11T05:00:58.887 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:00:58.887 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:00:58.893 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.4 MB/s | 4.8 MB 00:00 2024-01-11T05:00:58.918 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 391 kB/s | 49 kB 00:00 2024-01-11T05:00:58.943 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 967 kB/s | 355 kB 00:00 2024-01-11T05:00:58.943 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:00:58.944 INFO:teuthology.orchestra.run.smithi067.stdout:Total 5.8 MB/s | 5.8 MB 00:01 2024-01-11T05:00:58.997 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:00:59.007 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:00:59.008 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:00:59.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-01-11T05:00:59.115 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:00:59.119 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:00:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:00:59.642 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-11T05:00:59.920 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:12 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:00.042 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-11T05:01:01.034 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-11T05:01:01.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-11T05:01:01.638 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-11T05:01:01.645 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:01.646 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:01.646 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-01-11T05:01:01.646 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:01.646 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:01:01.646 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 911 k 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 994 k 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.8 M 2024-01-11T05:01:01.647 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:01:01.828 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 6/6 2024-01-11T05:01:02.120 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 202 kB/s | 83 kB 00:00 2024-01-11T05:01:02.154 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-16.2.14-417.gcda00bbf.el8.x86_ 2.0 MB/s | 911 kB 00:00 2024-01-11T05:01:02.154 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:02.154 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.9 MB/s | 994 kB 00:00 2024-01-11T05:01:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 6/6 2024-01-11T05:01:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 1/6 2024-01-11T05:01:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-11T05:01:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-11T05:01:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-11T05:01:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-11T05:01:02.166 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:01:02.187 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:01:02.187 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:01:02.275 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:01:02.278 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:01:02.451 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:01:02.728 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-11T05:01:02.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:16.2.14-417.gcda00bbf.el8.noarch 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: python3-apipkg-1.5-6.el8.noarch 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:02.889 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:02.920 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 2/2 2024-01-11T05:01:03.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-01-11T05:01:03.342 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 2/2 2024-01-11T05:01:03.343 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/2 2024-01-11T05:01:03.924 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:13 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:04.041 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-11T05:01:04.041 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:04.041 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:01:04.041 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-11T05:01:04.041 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:04.041 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:04.229 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-01-11T05:01:05.098 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:18 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 911 k 2024-01-11T05:01:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-01-11T05:01:05.769 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-11T05:01:05.769 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:05.769 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:01:05.769 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:05.769 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-01-11T05:01:05.769 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:05.770 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 994 k 2024-01-11T05:01:05.770 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.8 M 2024-01-11T05:01:05.770 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:01:06.221 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): ceph-fuse-16.2.14-417.gcda00bbf.el8.x86_ 2.2 MB/s | 911 kB 00:00 2024-01-11T05:01:06.238 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 194 kB/s | 83 kB 00:00 2024-01-11T05:01:06.239 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:06.239 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.1 MB/s | 994 kB 00:00 2024-01-11T05:01:06.251 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:01:06.272 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:01:06.272 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:01:06.366 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:01:06.369 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:01:06.564 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:01:06.824 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-11T05:01:06.832 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:06.832 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 131 k 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:01:06.833 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:06.834 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 131 k 2024-01-11T05:01:06.834 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 432 k 2024-01-11T05:01:06.834 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:01:07.011 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 2/2 2024-01-11T05:01:07.134 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-16.2.14-417.gcda00bbf.el8.x86_64 437 kB/s | 131 kB 00:00 2024-01-11T05:01:07.135 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:07.135 INFO:teuthology.orchestra.run.smithi169.stdout:Total 436 kB/s | 131 kB 00:00 2024-01-11T05:01:07.135 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:01:07.140 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:01:07.140 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:01:07.191 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:01:07.194 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:01:07.317 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:01:07.421 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 2/2 2024-01-11T05:01:07.421 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/2 2024-01-11T05:01:07.569 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:07.860 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-11T05:01:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:08.197 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-11T05:01:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:08.198 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:08.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-01-11T05:01:08.582 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:08.582 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:08.582 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:01:08.582 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:08.583 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:08.583 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:08.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-01-11T05:01:09.305 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:19 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:09.659 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:22 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:11.142 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 131 k 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:11.143 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:01:11.144 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:11.144 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:01:11.144 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:11.144 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:11.145 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 131 k 2024-01-11T05:01:11.145 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 432 k 2024-01-11T05:01:11.145 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:01:11.334 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-16.2.14-417.gcda00bbf.el8.x86_64 690 kB/s | 131 kB 00:00 2024-01-11T05:01:11.335 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:11.335 INFO:teuthology.orchestra.run.smithi067.stdout:Total 686 kB/s | 131 kB 00:00 2024-01-11T05:01:11.335 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:01:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:01:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:01:11.381 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:11.383 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:11.383 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:11.397 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:01:11.401 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:01:11.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-01-11T05:01:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:01:11.726 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:12.021 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:12.358 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:25 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:12.760 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:12.761 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:12.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-01-11T05:01:13.882 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:23 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:14.068 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 25 k 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:01:14.069 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 25 k 2024-01-11T05:01:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 107 k 2024-01-11T05:01:14.070 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:01:14.222 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-16.2.14-417.gcda00bbf.el8.x86_6 168 kB/s | 25 kB 00:00 2024-01-11T05:01:14.222 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:14.223 INFO:teuthology.orchestra.run.smithi169.stdout:Total 166 kB/s | 25 kB 00:00 2024-01-11T05:01:14.223 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:01:14.226 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:01:14.226 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:01:14.229 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:01:14.232 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:01:14.283 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:01:14.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:14.783 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:15.467 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:15.482 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:15.482 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:15.482 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:01:15.482 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:15.482 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:15.482 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:15.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-01-11T05:01:15.722 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:15.723 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:15.723 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:15.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-01-11T05:01:16.466 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:29 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:16.717 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:26 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:17.965 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:18.200 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:18.201 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:18.201 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:18.306 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-01-11T05:01:18.547 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 25 k 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:01:18.548 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:18.549 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 25 k 2024-01-11T05:01:18.549 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 107 k 2024-01-11T05:01:18.549 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:01:18.690 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-16.2.14-417.gcda00bbf.el8.x86_6 182 kB/s | 25 kB 00:00 2024-01-11T05:01:18.690 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:18.690 INFO:teuthology.orchestra.run.smithi067.stdout:Total 180 kB/s | 25 kB 00:00 2024-01-11T05:01:18.691 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:01:18.694 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:01:18.694 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:01:18.696 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:01:18.700 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:01:18.746 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:01:18.967 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:19.160 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:32 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:19.241 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:19.947 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:19.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:19.947 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:19.947 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:19.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:19.947 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:20.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-01-11T05:01:20.637 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:20.876 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:20.877 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:20.877 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:20.960 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:30 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:20.988 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-01-11T05:01:21.865 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:34 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:22.536 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:22.790 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:22.792 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:22.792 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:22.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-01-11T05:01:23.351 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:23.587 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:23.588 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:23.588 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:23.696 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-01-11T05:01:23.790 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:33 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:24.567 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:37 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:25.389 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:25.645 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:25.646 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:25.646 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:25.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-01-11T05:01:26.043 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:26.278 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:26.280 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:26.280 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:26.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-01-11T05:01:26.641 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:36 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:27.261 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:40 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:28.219 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:28.472 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:28.474 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:28.474 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:28.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-01-11T05:01:28.731 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:28.966 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:28.967 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:28.967 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:29.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-01-11T05:01:29.479 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:39 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:29.934 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:42 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:31.052 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:31.304 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:31.306 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:31.306 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:31.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-01-11T05:01:31.418 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:31.657 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:32.115 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-01-11T05:01:32.115 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:32.116 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-01-11T05:01:32.308 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:42 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:32.994 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:45 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:33.880 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:34.132 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:34.133 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:34.133 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:34.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-01-11T05:01:34.713 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:34.713 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 88 k 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:01:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:34.715 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 88 k 2024-01-11T05:01:34.715 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 240 k 2024-01-11T05:01:34.715 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:01:34.996 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-16.2.14-417.gcda00bbf.el8.x86_64.rpm 314 kB/s | 88 kB 00:00 2024-01-11T05:01:34.997 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:34.997 INFO:teuthology.orchestra.run.smithi169.stdout:Total 313 kB/s | 88 kB 00:00 2024-01-11T05:01:34.997 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:01:35.007 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:01:35.007 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:01:35.057 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:01:35.060 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:01:35.127 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:45 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:35.208 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:01:35.389 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:35.679 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:36.356 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:36.357 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:36.357 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:01:36.357 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:36.357 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:36.357 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:36.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-01-11T05:01:36.729 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:16.2.14-417.gcda00bbf.el8.x86_64 is already installed. 2024-01-11T05:01:36.981 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:36.982 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-01-11T05:01:36.982 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:37.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-01-11T05:01:37.442 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:50 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:37.986 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:47 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:39.177 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 3.7 M 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:01:39.178 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:39.179 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.7 M 2024-01-11T05:01:39.179 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 15 M 2024-01-11T05:01:39.179 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:01:39.846 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:39.846 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 88 k 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 88 k 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 240 k 2024-01-11T05:01:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:01:40.338 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-16.2.14-417.gcda00bbf.el8.x86_64.rpm 180 kB/s | 88 kB 00:00 2024-01-11T05:01:40.338 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:40.339 INFO:teuthology.orchestra.run.smithi067.stdout:Total 180 kB/s | 88 kB 00:00 2024-01-11T05:01:40.339 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:01:40.349 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:01:40.349 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:01:40.404 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:01:40.407 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:01:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:01:40.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:40.991 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:41.257 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-16.2.14-417.gcda00bbf.el8.x86_64.rpm 1.8 MB/s | 3.7 MB 00:02 2024-01-11T05:01:41.258 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:41.258 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.8 MB/s | 3.7 MB 00:02 2024-01-11T05:01:41.259 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:01:41.272 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:01:41.272 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:01:41.339 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:01:41.342 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:01:41.737 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:41.737 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:41.737 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:41.737 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:41.737 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:41.737 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:41.779 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:01:41.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-01-11T05:01:41.997 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:42.418 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:42.810 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:52 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:43.143 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:43.144 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:43.144 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:01:43.144 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:43.144 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:43.144 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:43.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-01-11T05:01:44.250 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:57 ago on Thu 11 Jan 2024 04:58:47 AM UTC. 2024-01-11T05:01:44.648 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:44.648 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-01-11T05:01:44.648 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 3.7 M 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:01:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:44.650 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.7 M 2024-01-11T05:01:44.650 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 15 M 2024-01-11T05:01:44.650 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:01:45.681 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-16.2.14-417.gcda00bbf.el8.x86_64.rpm 3.6 MB/s | 3.7 MB 00:01 2024-01-11T05:01:45.682 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:45.682 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.6 MB/s | 3.7 MB 00:01 2024-01-11T05:01:45.683 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:01:45.696 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:01:45.696 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:01:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:01:45.769 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:01:45.956 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 175 k 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-01-11T05:01:45.957 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-01-11T05:01:45.958 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:45.958 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 175 k 2024-01-11T05:01:45.958 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 523 k 2024-01-11T05:01:45.958 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-01-11T05:01:46.159 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-16.2.14-417.gcda00bbf.el8.x86_64.rpm 872 kB/s | 175 kB 00:00 2024-01-11T05:01:46.160 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:46.160 INFO:teuthology.orchestra.run.smithi169.stdout:Total 867 kB/s | 175 kB 00:00 2024-01-11T05:01:46.160 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-01-11T05:01:46.171 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-01-11T05:01:46.171 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-01-11T05:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-01-11T05:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-01-11T05:01:46.236 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:01:46.340 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-01-11T05:01:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:46.554 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:46.825 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:46.852 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:47.565 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:47.565 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:47.565 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-01-11T05:01:47.565 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:47.565 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:01:47.565 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-01-11T05:01:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:47.576 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:47.577 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:47.577 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:47.577 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:47.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-01-11T05:01:47.761 DEBUG:teuthology.parallel:result is None 2024-01-11T05:01:48.653 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:58 ago on Thu 11 Jan 2024 04:58:50 AM UTC. 2024-01-11T05:01:50.496 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-01-11T05:01:50.497 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:50.497 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-01-11T05:01:50.497 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:50.497 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-01-11T05:01:50.497 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:16.2.14-417.gcda00bbf.el8 ceph 175 k 2024-01-11T05:01:50.497 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 175 k 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 523 k 2024-01-11T05:01:50.498 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-01-11T05:01:50.692 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-16.2.14-417.gcda00bbf.el8.x86_64.rpm 905 kB/s | 175 kB 00:00 2024-01-11T05:01:50.693 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-01-11T05:01:50.693 INFO:teuthology.orchestra.run.smithi067.stdout:Total 899 kB/s | 175 kB 00:00 2024-01-11T05:01:50.693 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-01-11T05:01:50.708 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-01-11T05:01:50.708 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-01-11T05:01:50.759 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-01-11T05:01:50.762 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-01-11T05:01:50.883 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-01-11T05:01:51.055 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:51.388 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:52.108 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 1/1 2024-01-11T05:01:52.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:52.109 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-01-11T05:01:52.109 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:16.2.14-417.gcda00bbf.el8.x86_64 2024-01-11T05:01:52.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:01:52.109 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-01-11T05:01:52.286 DEBUG:teuthology.parallel:result is None 2024-01-11T05:01:52.286 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T05:01:52.286 INFO:teuthology.packaging:ref: None 2024-01-11T05:01:52.286 INFO:teuthology.packaging:tag: None 2024-01-11T05:01:52.286 INFO:teuthology.packaging:branch: pacific 2024-01-11T05:01:52.286 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T05:01:52.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-01-11T05:01:52.515 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-11T05:01:52.538 INFO:teuthology.orchestra.run.smithi067.stdout:16.2.14-417.gcda00bbf.el8 2024-01-11T05:01:52.538 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-417.gcda00bbf.el8 2024-01-11T05:01:52.538 INFO:teuthology.task.install:The correct ceph version 16.2.14-417.gcda00bbf is installed. 2024-01-11T05:01:52.543 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T05:01:52.543 INFO:teuthology.packaging:ref: None 2024-01-11T05:01:52.543 INFO:teuthology.packaging:tag: None 2024-01-11T05:01:52.543 INFO:teuthology.packaging:branch: pacific 2024-01-11T05:01:52.543 INFO:teuthology.packaging:sha1: 4bcedb9fc11bed1fd6fd0c4fd59187401f309680 2024-01-11T05:01:52.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-01-11T05:01:52.765 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-11T05:01:52.792 INFO:teuthology.orchestra.run.smithi169.stdout:16.2.14-417.gcda00bbf.el8 2024-01-11T05:01:52.793 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-417.gcda00bbf.el8 2024-01-11T05:01:52.793 INFO:teuthology.task.install:The correct ceph version 16.2.14-417.gcda00bbf is installed. 2024-01-11T05:01:52.794 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-11T05:01:52.795 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:01:52.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T05:01:52.824 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T05:01:52.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T05:01:52.866 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-11T05:01:52.866 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:01:52.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T05:01:52.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T05:01:52.961 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T05:01:52.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T05:01:52.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T05:01:53.062 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-11T05:01:53.062 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:01:53.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T05:01:53.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T05:01:53.157 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T05:01:53.157 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T05:01:53.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T05:01:53.257 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-11T05:01:53.257 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:01:53.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T05:01:53.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T05:01:53.352 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T05:01:53.352 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T05:01:53.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T05:01:53.449 INFO:teuthology.run_tasks:Running task print... 2024-01-11T05:01:53.459 INFO:teuthology.task.print:**** done install task... 2024-01-11T05:01:53.459 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-11T05:01:53.581 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '4bcedb9fc11bed1fd6fd0c4fd59187401f309680'} 2024-01-11T05:01:53.581 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2024-01-11T05:01:53.581 INFO:tasks.cephadm:Cluster fsid is 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:01:53.582 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-11T05:01:53.582 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-11T05:01:53.582 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi067': '172.21.15.67', 'mon.smithi169': '172.21.15.169'} 2024-01-11T05:01:53.582 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-11T05:01:53.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-01-11T05:01:53.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-01-11T05:01:53.640 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2024-01-11T05:01:53.641 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-11T05:01:53.889 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 11 05:01 /home/ubuntu/cephtest/cephadm 2024-01-11T05:01:53.889 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-11T05:01:54.027 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 11 05:01 /home/ubuntu/cephtest/cephadm 2024-01-11T05:01:54.028 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-11T05:01:54.049 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-11T05:01:54.078 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2024-01-11T05:01:54.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2024-01-11T05:01:54.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2024-01-11T05:01:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-01-11T05:01:55.277 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-01-11T05:02:19.090 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-01-11T05:02:22.805 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-01-11T05:02:22.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-01-11T05:02:22.863 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-01-11T05:02:22.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-01-11T05:02:22.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-01-11T05:02:22.961 INFO:tasks.cephadm:Writing seed config... 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-11T05:02:22.962 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-11T05:02:22.963 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-11T05:02:22.964 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-11T05:02:22.965 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-11T05:02:22.965 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:02:22.965 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-11T05:02:22.994 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 89773ea0-b03e-11ee-95ab-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-11T05:02:22.994 DEBUG:teuthology.orchestra.run.smithi067:mon.smithi067> sudo journalctl -f -n 0 -u ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067.service 2024-01-11T05:02:23.036 INFO:tasks.cephadm:Bootstrapping... 2024-01-11T05:02:23.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-11T05:02:23.066 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:-- Logs begin at Thu 2024-01-11 04:51:43 UTC. -- 2024-01-11T05:02:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:-------------------------------------------------------------------------------- 2024-01-11T05:02:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', '89773ea0-b03e-11ee-95ab-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.67', '--skip-admin-label'] 2024-01-11T05:02:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 4.6.1 2024-01-11T05:02:23.288 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying podman|docker is present... 2024-01-11T05:02:23.308 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 4.6.1 2024-01-11T05:02:23.318 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying lvm2 is present... 2024-01-11T05:02:23.318 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying time synchronization is in place... 2024-01-11T05:02:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-11T05:02:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-01-11T05:02:23.337 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: enabled 2024-01-11T05:02:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: active 2024-01-11T05:02:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:Unit chronyd.service is enabled and running 2024-01-11T05:02:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:Repeating the final host check... 2024-01-11T05:02:23.363 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 4.6.1 2024-01-11T05:02:23.375 INFO:teuthology.orchestra.run.smithi067.stderr:podman (/bin/podman) version 4.6.1 is present 2024-01-11T05:02:23.375 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl is present 2024-01-11T05:02:23.375 INFO:teuthology.orchestra.run.smithi067.stderr:lvcreate is present 2024-01-11T05:02:23.381 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-11T05:02:23.386 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-01-11T05:02:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: enabled 2024-01-11T05:02:23.398 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: active 2024-01-11T05:02:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:Unit chronyd.service is enabled and running 2024-01-11T05:02:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:Host looks OK 2024-01-11T05:02:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:Cluster fsid: 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:23.399 INFO:teuthology.orchestra.run.smithi067.stderr:Acquiring lock 140231896009192 on /run/cephadm/89773ea0-b03e-11ee-95ab-87774f69a715.lock 2024-01-11T05:02:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:Lock 140231896009192 acquired on /run/cephadm/89773ea0-b03e-11ee-95ab-87774f69a715.lock 2024-01-11T05:02:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying IP 172.21.15.67 port 3300 ... 2024-01-11T05:02:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying IP 172.21.15.67 port 6789 ... 2024-01-11T05:02:23.400 INFO:teuthology.orchestra.run.smithi067.stderr:Base mon IP(s) is [172.21.15.67:3300, 172.21.15.67:6789], mon addrv is [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-01-11T05:02:23.403 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.67 metric 100 2024-01-11T05:02:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.67 metric 100 2024-01-11T05:02:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-11T05:02:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-11T05:02:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-01-11T05:02:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-11T05:02:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-11T05:02:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-11T05:02:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-11T05:02:23.409 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-01-11T05:02:23.410 INFO:teuthology.orchestra.run.smithi067.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-11T05:02:23.410 INFO:teuthology.orchestra.run.smithi067.stderr:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-01-11T05:02:23.410 INFO:teuthology.orchestra.run.smithi067.stderr:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-01-11T05:02:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-11T05:02:23.411 INFO:teuthology.orchestra.run.smithi067.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-11T05:02:23.412 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2024-01-11T05:02:23.491 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2024-01-11T05:02:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Getting image source signatures 2024-01-11T05:02:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:a997af56d24c50f31dbc8bd557a1c2adeb9ea675d7d7142f1a37cb629070c246 2024-01-11T05:02:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-11T05:02:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Copying config sha256:46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2024-01-11T05:02:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: Writing manifest to image destination 2024-01-11T05:02:30.034 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/podman: 46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2024-01-11T05:02:30.422 INFO:teuthology.orchestra.run.smithi067.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2024-01-11T05:02:30.764 INFO:teuthology.orchestra.run.smithi067.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2024-01-11T05:02:30.764 INFO:teuthology.orchestra.run.smithi067.stderr:Extracting ceph user uid/gid from container image... 2024-01-11T05:02:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:stat: 167 167 2024-01-11T05:02:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:Creating initial keys... 2024-01-11T05:02:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-authtool: AQBndp9l8EXJJRAAuLjUcocDO3RWvZKhnMY5Tg== 2024-01-11T05:02:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-authtool: AQBodp9lhxg3DxAAX811nAA8INVtGZjLfHq+6Q== 2024-01-11T05:02:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-authtool: AQBodp9lg3msNBAA8u2QJmMPbHqdXL7uxW1IgA== 2024-01-11T05:02:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:Creating initial monmap... 2024-01-11T05:02:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-11T05:02:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-11T05:02:33.847 INFO:teuthology.orchestra.run.smithi067.stderr:monmaptool for smithi067 [v2:172.21.15.67:3300,v1:172.21.15.67:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-11T05:02:33.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: set fsid to 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:33.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-11T05:02:33.847 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:02:33.847 INFO:teuthology.orchestra.run.smithi067.stderr:Creating mon... 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.107+0000 7f02e44c8700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.107+0000 7f02e44c8700 1 imported monmap: 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: fsid 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: last_changed 2024-01-11T05:02:33.494211+0000 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: created 2024-01-11T05:02:33.494211+0000 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.smithi067 2024-01-11T05:02:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.107+0000 7f02e44c8700 0 /usr/bin/ceph-mon: set fsid to 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: RocksDB version: 6.8.1 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Compile date Jul 8 2021 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: DB SUMMARY 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi067/store.db dir, Total Num: 0, files: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi067/store.db: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.error_if_exists: 0 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.create_if_missing: 1 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.paranoid_checks: 1 2024-01-11T05:02:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.env: 0x560d1e20b1c0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.fs: Posix File System 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.info_log: 0x560d1fb95000 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.statistics: (nil) 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.use_fsync: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_log_file_size: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.allow_fallocate: 1 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.use_direct_reads: 0 2024-01-11T05:02:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.db_log_dir: 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi067/store.db 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_subcompactions: 1 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_background_flushes: -1 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-11T05:02:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.write_buffer_manager: 0x560d1fb9d530 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.unordered_write: 0 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-11T05:02:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.row_cache: None 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.wal_filter: None 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.preserve_deletes: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.two_write_queues: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.atomic_flush: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.log_readahead_size: 0 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_background_jobs: 2 2024-01-11T05:02:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_background_compactions: -1 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.max_open_files: -1 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-11T05:02:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Compression algorithms supported: 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kZSTD supported: 0 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kXpressCompression supported: 0 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kLZ4Compression supported: 1 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kBZip2Compression supported: 0 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kZlibCompression supported: 1 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: kSnappyCompression supported: 1 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000001 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.merge_operator: 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.compaction_filter: None 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d1faae160) 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-11T05:02:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_cache: 0x560d1fae4d10 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-11T05:02:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.108+0000 7f02e44c8700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-11T05:02:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression: NoCompression 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.num_levels: 7 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-11T05:02:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-11T05:02:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-11T05:02:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-11T05:02:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.table_properties_collectors: 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.inplace_update_support: 0 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-11T05:02:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.bloom_locality: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.max_successive_merges: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-11T05:02:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.ttl: 2592000 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.109+0000 7f02e44c8700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.110+0000 7f02e44c8700 4 rocksdb: DB pointer 0x560d1fba9800 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.110+0000 7f02cd351700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.110+0000 7f02cd351700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-11T05:02:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-11T05:02:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-11T05:02:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.111+0000 7f02e44c8700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.111+0000 7f02e44c8700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-11T05:02:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph-mon: debug 2024-01-11T05:02:34.111+0000 7f02e44c8700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi067 for mon.smithi067 2024-01-11T05:02:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:create mon.smithi067 on 2024-01-11T05:02:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-11T05:02:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-11T05:02:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-89773ea0-b03e-11ee-95ab-87774f69a715.target → /etc/systemd/system/ceph-89773ea0-b03e-11ee-95ab-87774f69a715.target. 2024-01-11T05:02:34.691 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-89773ea0-b03e-11ee-95ab-87774f69a715.target → /etc/systemd/system/ceph-89773ea0-b03e-11ee-95ab-87774f69a715.target. 2024-01-11T05:02:34.901 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to reset failed state of unit ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067.service: Unit ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067.service not loaded. 2024-01-11T05:02:34.908 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/ceph-89773ea0-b03e-11ee-95ab-87774f69a715.target.wants/ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067.service → /etc/systemd/system/ceph-89773ea0-b03e-11ee-95ab-87774f69a715@.service. 2024-01-11T05:02:35.343 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-01-11T05:02:35.351 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-01-11T05:02:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-01-11T05:02:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to enable service . firewalld.service is not available 2024-01-11T05:02:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mon to start... 2024-01-11T05:02:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mon... 2024-01-11T05:02:35.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: cluster 2024-01-11T05:02:35.347396+0000 mon.smithi067 (mon.0) 1 : cluster [INF] mon.smithi067 is new leader, mons smithi067 in quorum (ranks 0) 2024-01-11T05:02:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.793+0000 7fc174ed8700 1 Processor -- start 2024-01-11T05:02:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.793+0000 7fc174ed8700 1 -- start start 2024-01-11T05:02:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.793+0000 7fc174ed8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc1700fc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.793+0000 7fc174ed8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1700fc880 con 0x7fc1700fbf70 2024-01-11T05:02:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc1700fc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc1700fc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43930/0 (socket says 172.21.15.67:43930) 2024-01-11T05:02:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16e59c700 1 -- 172.21.15.67:0/1272759726 learned_addr learned my addr 172.21.15.67:0/1272759726 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16e59c700 1 -- 172.21.15.67:0/1272759726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700fc9c0 con 0x7fc1700fbf70 2024-01-11T05:02:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16e59c700 1 --2- 172.21.15.67:0/1272759726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc1700fc390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc16800b5f0 tx=0x7fc168006ee0).ready entity=mon.0 client_cookie=fa4200b81d7671ac server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16d59a700 1 -- 172.21.15.67:0/1272759726 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc16800d600 con 0x7fc1700fbf70 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16d59a700 1 -- 172.21.15.67:0/1272759726 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16800d760 con 0x7fc1700fbf70 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.794+0000 7fc16d59a700 1 -- 172.21.15.67:0/1272759726 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc16800da30 con 0x7fc1700fbf70 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 -- 172.21.15.67:0/1272759726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 msgr2=0x7fc1700fc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 --2- 172.21.15.67:0/1272759726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc1700fc390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc16800b5f0 tx=0x7fc168006ee0).stop 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 -- 172.21.15.67:0/1272759726 shutdown_connections 2024-01-11T05:02:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 --2- 172.21.15.67:0/1272759726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc1700fc390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 -- 172.21.15.67:0/1272759726 >> 172.21.15.67:0/1272759726 conn(0x7fc1700f7770 msgr2=0x7fc1700f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 -- 172.21.15.67:0/1272759726 shutdown_connections 2024-01-11T05:02:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.795+0000 7fc174ed8700 1 -- 172.21.15.67:0/1272759726 wait complete. 2024-01-11T05:02:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc174ed8700 1 Processor -- start 2024-01-11T05:02:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc174ed8700 1 -- start start 2024-01-11T05:02:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc174ed8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc17010a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc174ed8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17010aa40 con 0x7fc1700fbf70 2024-01-11T05:02:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc16e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc17010a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc16e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc17010a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43938/0 (socket says 172.21.15.67:43938) 2024-01-11T05:02:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.796+0000 7fc16e59c700 1 -- 172.21.15.67:0/708415276 learned_addr learned my addr 172.21.15.67:0/708415276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc16e59c700 1 -- 172.21.15.67:0/708415276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16800b200 con 0x7fc1700fbf70 2024-01-11T05:02:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc16e59c700 1 --2- 172.21.15.67:0/708415276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc17010a550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc168006c70 tx=0x7fc168016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc168011b40 con 0x7fc1700fbf70 2024-01-11T05:02:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc168011ca0 con 0x7fc1700fbf70 2024-01-11T05:02:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17010ac40 con 0x7fc1700fbf70 2024-01-11T05:02:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc16801d430 con 0x7fc1700fbf70 2024-01-11T05:02:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17010b0d0 con 0x7fc1700fbf70 2024-01-11T05:02:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fc16800f070 con 0x7fc1700fbf70 2024-01-11T05:02:35.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.797+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc16801dc10 con 0x7fc1700fbf70 2024-01-11T05:02:35.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.798+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170045bb0 con 0x7fc1700fbf70 2024-01-11T05:02:35.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.800+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fc168016070 con 0x7fc1700fbf70 2024-01-11T05:02:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.869+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc170103fd0 con 0x7fc1700fbf70 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.869+0000 7fc166ffd700 1 -- 172.21.15.67:0/708415276 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fc16801d6a0 con 0x7fc1700fbf70 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: cluster: 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: id: 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: services: 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi067 (age 0.500173s) 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: data: 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-11T05:02:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: pgs: 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 msgr2=0x7fc17010a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 --2- 172.21.15.67:0/708415276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc17010a550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc168006c70 tx=0x7fc168016000).stop 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 shutdown_connections 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 --2- 172.21.15.67:0/708415276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700fbf70 0x7fc17010a550 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 >> 172.21.15.67:0/708415276 conn(0x7fc1700f7770 msgr2=0x7fc1700fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 shutdown_connections 2024-01-11T05:02:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:35.870+0000 7fc174ed8700 1 -- 172.21.15.67:0/708415276 wait complete. 2024-01-11T05:02:36.224 INFO:teuthology.orchestra.run.smithi067.stderr:mon is available 2024-01-11T05:02:36.224 INFO:teuthology.orchestra.run.smithi067.stderr:Assimilating anything we can from ceph.conf... 2024-01-11T05:02:36.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: cluster 2024-01-11T05:02:35.371206+0000 mon.smithi067 (mon.0) 2 : cluster [INF] mon.smithi067 is new leader, mons smithi067 in quorum (ranks 0) 2024-01-11T05:02:36.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: cluster 2024-01-11T05:02:35.371460+0000 mon.smithi067 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} 2024-01-11T05:02:36.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: cluster 2024-01-11T05:02:35.375032+0000 mon.smithi067 (mon.0) 4 : cluster [DBG] fsmap 2024-01-11T05:02:36.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: cluster 2024-01-11T05:02:35.379192+0000 mon.smithi067 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-11T05:02:36.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: cluster 2024-01-11T05:02:35.379605+0000 mon.smithi067 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-11T05:02:36.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: audit 2024-01-11T05:02:35.871332+0000 mon.smithi067 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.67:0/708415276' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-11T05:02:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.639+0000 7f387fd1b700 1 Processor -- start 2024-01-11T05:02:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.639+0000 7f387fd1b700 1 -- start start 2024-01-11T05:02:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.639+0000 7f387fd1b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.639+0000 7f387fd1b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38780ff1b0 con 0x7f3878027590 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387dab7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387dab7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43944/0 (socket says 172.21.15.67:43944) 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387dab7700 1 -- 172.21.15.67:0/1316749656 learned_addr learned my addr 172.21.15.67:0/1316749656 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387dab7700 1 -- 172.21.15.67:0/1316749656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780ff2f0 con 0x7f3878027590 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387dab7700 1 --2- 172.21.15.67:0/1316749656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fecc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f386800b5f0 tx=0x7f3868006ee0).ready entity=mon.0 client_cookie=53f397ed6b29cfdd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387cab5700 1 -- 172.21.15.67:0/1316749656 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f386800d600 con 0x7f3878027590 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387cab5700 1 -- 172.21.15.67:0/1316749656 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386800d760 con 0x7f3878027590 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.640+0000 7f387cab5700 1 -- 172.21.15.67:0/1316749656 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f386800da30 con 0x7f3878027590 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 -- 172.21.15.67:0/1316749656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 msgr2=0x7f38780fecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 --2- 172.21.15.67:0/1316749656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fecc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f386800b5f0 tx=0x7f3868006ee0).stop 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 -- 172.21.15.67:0/1316749656 shutdown_connections 2024-01-11T05:02:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 --2- 172.21.15.67:0/1316749656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fecc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 -- 172.21.15.67:0/1316749656 >> 172.21.15.67:0/1316749656 conn(0x7f38780f6680 msgr2=0x7f38780f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 -- 172.21.15.67:0/1316749656 shutdown_connections 2024-01-11T05:02:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 -- 172.21.15.67:0/1316749656 wait complete. 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.641+0000 7f387fd1b700 1 Processor -- start 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387fd1b700 1 -- start start 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387fd1b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387fd1b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38780fabc0 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387dab7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387dab7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43956/0 (socket says 172.21.15.67:43956) 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387dab7700 1 -- 172.21.15.67:0/535174358 learned_addr learned my addr 172.21.15.67:0/535174358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387dab7700 1 -- 172.21.15.67:0/535174358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386800b200 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387dab7700 1 --2- 172.21.15.67:0/535174358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fc740 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3868006c70 tx=0x7f3868016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3868011b40 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3868011ca0 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38780fad60 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.642+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f386801d430 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.643+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38780fb1f0 con 0x7f3878027590 2024-01-11T05:02:36.647 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.643+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f386800f070 con 0x7f3878027590 2024-01-11T05:02:36.648 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.643+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f386801dbf0 con 0x7f3878027590 2024-01-11T05:02:36.648 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.644+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3878045bb0 con 0x7f3878027590 2024-01-11T05:02:36.648 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.646+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3868016070 con 0x7f3878027590 2024-01-11T05:02:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.713+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f38780ff4e0 con 0x7f3878027590 2024-01-11T05:02:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.717+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f386801d5e0 con 0x7f3878027590 2024-01-11T05:02:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.717+0000 7f38667fc700 1 -- 172.21.15.67:0/535174358 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7f3868036020 con 0x7f3878027590 2024-01-11T05:02:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [global] 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: fsid = 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: log_to_journald = false 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [mgr] 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-11T05:02:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [osd] 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 msgr2=0x7f38780fc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 --2- 172.21.15.67:0/535174358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fc740 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3868006c70 tx=0x7f3868016000).stop 2024-01-11T05:02:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 shutdown_connections 2024-01-11T05:02:36.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 --2- 172.21.15.67:0/535174358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3878027590 0x7f38780fc740 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:36.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 >> 172.21.15.67:0/535174358 conn(0x7f38780f6680 msgr2=0x7f38780f8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:36.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 shutdown_connections 2024-01-11T05:02:36.725 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:36.718+0000 7f387fd1b700 1 -- 172.21.15.67:0/535174358 wait complete. 2024-01-11T05:02:37.081 INFO:teuthology.orchestra.run.smithi067.stderr:Generating new minimal ceph.conf... 2024-01-11T05:02:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.438+0000 7f0a98cdb700 1 Processor -- start 2024-01-11T05:02:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.438+0000 7f0a98cdb700 1 -- start start 2024-01-11T05:02:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a98cdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a940fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a98cdb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940fb7e0 con 0x7f0a940faed0 2024-01-11T05:02:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a9259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a940fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a9259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a940fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43972/0 (socket says 172.21.15.67:43972) 2024-01-11T05:02:37.442 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a9259c700 1 -- 172.21.15.67:0/2485121888 learned_addr learned my addr 172.21.15.67:0/2485121888 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:37.442 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a9259c700 1 -- 172.21.15.67:0/2485121888 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a940fb920 con 0x7f0a940faed0 2024-01-11T05:02:37.442 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.439+0000 7f0a9259c700 1 --2- 172.21.15.67:0/2485121888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a940fb2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0a8c009960 tx=0x7f0a8c0092d0).ready entity=mon.0 client_cookie=ec8f70f54847ca9b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:37.442 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a9159a700 1 -- 172.21.15.67:0/2485121888 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a8c00b9c0 con 0x7f0a940faed0 2024-01-11T05:02:37.442 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a9159a700 1 -- 172.21.15.67:0/2485121888 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0a8c011400 con 0x7f0a940faed0 2024-01-11T05:02:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a9159a700 1 -- 172.21.15.67:0/2485121888 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a8c010580 con 0x7f0a940faed0 2024-01-11T05:02:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a98cdb700 1 -- 172.21.15.67:0/2485121888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 msgr2=0x7f0a940fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a98cdb700 1 --2- 172.21.15.67:0/2485121888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a940fb2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0a8c009960 tx=0x7f0a8c0092d0).stop 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a98cdb700 1 -- 172.21.15.67:0/2485121888 shutdown_connections 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a98cdb700 1 --2- 172.21.15.67:0/2485121888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a940fb2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.440+0000 7f0a98cdb700 1 -- 172.21.15.67:0/2485121888 >> 172.21.15.67:0/2485121888 conn(0x7f0a940f6680 msgr2=0x7f0a940f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.441+0000 7f0a98cdb700 1 -- 172.21.15.67:0/2485121888 shutdown_connections 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.441+0000 7f0a98cdb700 1 -- 172.21.15.67:0/2485121888 wait complete. 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.441+0000 7f0a98cdb700 1 Processor -- start 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.441+0000 7f0a98cdb700 1 -- start start 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a98cdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a94109ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a98cdb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9410a1d0 con 0x7f0a940faed0 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a9259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a94109ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a9259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a94109ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43988/0 (socket says 172.21.15.67:43988) 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a9259c700 1 -- 172.21.15.67:0/3407910574 learned_addr learned my addr 172.21.15.67:0/3407910574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a9259c700 1 -- 172.21.15.67:0/3407910574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c009570 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a9259c700 1 --2- 172.21.15.67:0/3407910574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a94109ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0a8c011aa0 tx=0x7f0a8c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a8c010840 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0a8c010e00 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9410a3d0 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.442+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a8c01bd30 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.443+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f0a8c00f070 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.443+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9410af80 con 0x7f0a940faed0 2024-01-11T05:02:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.443+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0a8c01a3a0 con 0x7f0a940faed0 2024-01-11T05:02:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.444+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94045bb0 con 0x7f0a940faed0 2024-01-11T05:02:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.446+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0a8c011e60 con 0x7f0a940faed0 2024-01-11T05:02:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.512+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0a940fcd10 con 0x7f0a940faed0 2024-01-11T05:02:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a8affd700 1 -- 172.21.15.67:0/3407910574 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f0a8c01b440 con 0x7f0a940faed0 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 msgr2=0x7f0a94109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a98cdb700 1 --2- 172.21.15.67:0/3407910574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a94109ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0a8c011aa0 tx=0x7f0a8c0092d0).stop 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 shutdown_connections 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a98cdb700 1 --2- 172.21.15.67:0/3407910574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a940faed0 0x7f0a94109ce0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 >> 172.21.15.67:0/3407910574 conn(0x7f0a940f6680 msgr2=0x7f0a940f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.513+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 shutdown_connections 2024-01-11T05:02:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:37.514+0000 7f0a98cdb700 1 -- 172.21.15.67:0/3407910574 wait complete. 2024-01-11T05:02:37.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: audit 2024-01-11T05:02:36.715613+0000 mon.smithi067 (mon.0) 8 : audit [INF] from='client.? 172.21.15.67:0/535174358' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-11T05:02:37.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: audit 2024-01-11T05:02:36.719293+0000 mon.smithi067 (mon.0) 9 : audit [INF] from='client.? 172.21.15.67:0/535174358' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-11T05:02:37.865 INFO:teuthology.orchestra.run.smithi067.stderr:Restarting the monitor... 2024-01-11T05:02:38.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:37 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for 89773ea0-b03e-11ee-95ab-87774f69a715... 2024-01-11T05:02:38.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:37 smithi067 bash[38752]: Error: no container with name or ID "ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon.smithi067" found: no such container 2024-01-11T05:02:38.347 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: debug 2024-01-11T05:02:38.071+0000 7fc518f49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-11T05:02:38.347 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38428]: debug 2024-01-11T05:02:38.071+0000 7fc518f49700 -1 mon.smithi067@0(leader) e1 *** Got Signal Terminated *** 2024-01-11T05:02:38.630 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 bash[38791]: ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067 2024-01-11T05:02:38.910 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 systemd[1]: ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067.service: Succeeded. 2024-01-11T05:02:38.910 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 systemd[1]: Stopped Ceph mon.smithi067 for 89773ea0-b03e-11ee-95ab-87774f69a715. 2024-01-11T05:02:38.910 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 systemd[1]: Starting Ceph mon.smithi067 for 89773ea0-b03e-11ee-95ab-87774f69a715... 2024-01-11T05:02:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.973+0000 7feea4ca4700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.973+0000 7feea4ca4700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.973+0000 7feea4ca4700 0 pidfile_write: ignore empty --pid-file 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 bash[38914]: ec1e06fe2ca52134a4afd80d06119d7b85fa28a7cf547ad50b0e3e303ac10c9b 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.978+0000 7feea4ca4700 0 load: jerasure load: lrc load: isa 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: RocksDB version: 6.8.1 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Compile date Jul 8 2021 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: DB SUMMARY 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: CURRENT file: CURRENT 2024-01-11T05:02:39.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi067/store.db dir, Total Num: 1, files: 000004.sst 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi067/store.db: 000006.log size: 94692 ; 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.error_if_exists: 0 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.create_if_missing: 0 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.paranoid_checks: 1 2024-01-11T05:02:39.264 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.env: 0x562e99a961c0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.fs: Posix File System 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.info_log: 0x562e9b9c3a40 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.statistics: (nil) 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.use_fsync: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_log_file_size: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.allow_fallocate: 1 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.use_direct_reads: 0 2024-01-11T05:02:39.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-11T05:02:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-11T05:02:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.db_log_dir: 2024-01-11T05:02:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi067/store.db 2024-01-11T05:02:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-11T05:02:39.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_subcompactions: 1 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_background_flushes: -1 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-11T05:02:39.268 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.write_buffer_manager: 0x562e9c63a030 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.unordered_write: 0 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-11T05:02:39.269 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.row_cache: None 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.wal_filter: None 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.preserve_deletes: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.two_write_queues: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.atomic_flush: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.log_readahead_size: 0 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_background_jobs: 2 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_background_compactions: -1 2024-01-11T05:02:39.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_open_files: -1 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-11T05:02:39.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Compression algorithms supported: 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kZSTD supported: 0 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kXpressCompression supported: 0 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kLZ4Compression supported: 1 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kBZip2Compression supported: 0 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kZlibCompression supported: 1 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: kSnappyCompression supported: 1 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000005 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.merge_operator: 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_filter: None 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562e9b8ddcc0) 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cache_index_and_filter_blocks: 1 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: pin_top_level_index_and_filter: 1 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: index_type: 0 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: data_block_index_type: 0 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: index_shortening: 1 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: data_block_hash_table_util_ratio: 0.750000 2024-01-11T05:02:39.273 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: hash_index_allow_collision: 1 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: checksum: 1 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: no_block_cache: 0 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_cache: 0x562e9b912f10 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_cache_name: BinnedLRUCache 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_cache_options: 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: capacity : 536870912 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: num_shard_bits : 4 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: strict_capacity_limit : 0 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: high_pri_pool_ratio: 0.000 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_cache_compressed: (nil) 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: persistent_cache: (nil) 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_size: 4096 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_size_deviation: 10 2024-01-11T05:02:39.274 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_restart_interval: 16 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: index_block_restart_interval: 1 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: metadata_block_size: 4096 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: partition_filters: 0 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: use_delta_encoding: 1 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: whole_key_filtering: 1 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: verify_compression: 0 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: read_amp_bytes_per_bit: 0 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: format_version: 2 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: enable_index_compression: 1 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: block_align: 0 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-11T05:02:39.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression: NoCompression 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.num_levels: 7 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-11T05:02:39.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-11T05:02:39.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-11T05:02:39.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.table_properties_collectors: 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.inplace_update_support: 0 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-11T05:02:39.279 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.bloom_locality: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.max_successive_merges: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.ttl: 2592000 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.979+0000 7feea4ca4700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.980+0000 7feea4ca4700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.980+0000 7feea4ca4700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 systemd[1]: Started Ceph mon.smithi067 for 89773ea0-b03e-11ee-95ab-87774f69a715. 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.980+0000 7feea4ca4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1704949358982053, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.980+0000 7feea4ca4700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.980+0000 7feea4ca4700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.981+0000 7feea4ca4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1704949358983055, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 90208, "table_properties": {"data_size": 88330, "index_size": 355, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12870, "raw_average_key_size": 50, "raw_value_size": 81282, "raw_average_value_size": 316, "num_data_blocks": 11, "num_entries": 257, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1704949358, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.981+0000 7feea4ca4700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.981+0000 7feea4ca4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1704949358983467, "job": 1, "event": "recovery_finished"} 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.982+0000 7feea4ca4700 4 rocksdb: DB pointer 0x562e9b9d7800 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.982+0000 7fee8baac700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.982+0000 7fee8baac700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** DB Stats ** 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-11T05:02:39.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-11T05:02:39.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-11T05:02:39.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-11T05:02:39.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-11T05:02:39.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** Compaction Stats [default] ** 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: L0 2/0 89.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Sum 2/0 89.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** Compaction Stats [default] ** 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(Total Files): cumulative 0, interval 0 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(Keys): cumulative 0, interval 0 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Cumulative compaction: 0.00 GB write, 27.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Interval compaction: 0.00 GB write, 27.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-11T05:02:39.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** File Read Latency Histogram By Level [default] ** 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** Compaction Stats [default] ** 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: L0 2/0 89.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Sum 2/0 89.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** Compaction Stats [default] ** 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.5 0.00 0.00 1 0.001 0 0 2024-01-11T05:02:39.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(Total Files): cumulative 0, interval 0 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: AddFile(Keys): cumulative 0, interval 0 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Cumulative compaction: 0.00 GB write, 27.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ** File Read Latency Histogram By Level [default] ** 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.983+0000 7feea4ca4700 0 starting mon.smithi067 rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi067 fsid 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 1 mon.smithi067@-1(???) e1 preinit fsid 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:39.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 0 mon.smithi067@-1(???).mds e1 new map 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 0 mon.smithi067@-1(???).mds e1 print_map 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: e1 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: legacy client fscid: -1 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: No filesystems configured 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 0 mon.smithi067@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 0 mon.smithi067@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 0 mon.smithi067@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 0 mon.smithi067@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-11T05:02:39.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:36.715613+0000 mon.smithi067 (mon.0) 8 : audit [INF] from='client.? 172.21.15.67:0/535174358' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:36.719293+0000 mon.smithi067 (mon.0) 9 : audit [INF] from='client.? 172.21.15.67:0/535174358' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.984+0000 7feea4ca4700 1 mon.smithi067@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expand map: {default=false} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta from 'false' to 'false' 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expanded map: {default=false} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expand map: {default=info} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta from 'info' to 'info' 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expanded map: {default=info} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expand map: {default=daemon} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expand map: {default=debug} 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-11T05:02:39.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:02:38.985+0000 7feea4ca4700 20 expand_channel_meta expanded map: {default=debug} 2024-01-11T05:02:39.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:39.003415+0000 mon.smithi067 (mon.0) 1 : cluster [INF] mon.smithi067 is new leader, mons smithi067 in quorum (ranks 0) 2024-01-11T05:02:39.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:39.003498+0000 mon.smithi067 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} 2024-01-11T05:02:39.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:39.003542+0000 mon.smithi067 (mon.0) 3 : cluster [DBG] fsmap 2024-01-11T05:02:39.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:39.003575+0000 mon.smithi067 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-11T05:02:39.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:39.004028+0000 mon.smithi067 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-11T05:02:39.321 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.316+0000 7fbda5d20700 1 Processor -- start 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.316+0000 7fbda5d20700 1 -- start start 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.316+0000 7fbda5d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda00fc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.316+0000 7fbda5d20700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00fc8f0 con 0x7fbda00fbfe0 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda00fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda00fc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44002/0 (socket says 172.21.15.67:44002) 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9f7fe700 1 -- 172.21.15.67:0/3565809784 learned_addr learned my addr 172.21.15.67:0/3565809784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9f7fe700 1 -- 172.21.15.67:0/3565809784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00fca30 con 0x7fbda00fbfe0 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9f7fe700 1 --2- 172.21.15.67:0/3565809784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda00fc400 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fbd9400b5f0 tx=0x7fbd94006ee0).ready entity=mon.0 client_cookie=a9c22c046ea9c21f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9e7fc700 1 -- 172.21.15.67:0/3565809784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9400d600 con 0x7fbda00fbfe0 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9e7fc700 1 -- 172.21.15.67:0/3565809784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fbd9400dbc0 con 0x7fbda00fbfe0 2024-01-11T05:02:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.317+0000 7fbd9e7fc700 1 -- 172.21.15.67:0/3565809784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd94011b90 con 0x7fbda00fbfe0 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 -- 172.21.15.67:0/3565809784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 msgr2=0x7fbda00fc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 --2- 172.21.15.67:0/3565809784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda00fc400 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fbd9400b5f0 tx=0x7fbd94006ee0).stop 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 -- 172.21.15.67:0/3565809784 shutdown_connections 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 --2- 172.21.15.67:0/3565809784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda00fc400 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 -- 172.21.15.67:0/3565809784 >> 172.21.15.67:0/3565809784 conn(0x7fbda00f7790 msgr2=0x7fbda00f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 -- 172.21.15.67:0/3565809784 shutdown_connections 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.318+0000 7fbda5d20700 1 -- 172.21.15.67:0/3565809784 wait complete. 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbda5d20700 1 Processor -- start 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbda5d20700 1 -- start start 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbda5d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda010ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbda5d20700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda010b220 con 0x7fbda00fbfe0 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbd9f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda010ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbd9f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda010ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44012/0 (socket says 172.21.15.67:44012) 2024-01-11T05:02:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.319+0000 7fbd9f7fe700 1 -- 172.21.15.67:0/3355143462 learned_addr learned my addr 172.21.15.67:0/3355143462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd9f7fe700 1 -- 172.21.15.67:0/3355143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9400b200 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd9f7fe700 1 --2- 172.21.15.67:0/3355143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda010ad30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbd94000c00 tx=0x7fbd94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9401b570 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fbd9401bb30 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda010b420 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9401a740 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda010bfd0 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fbd9402b070 con 0x7fbda00fbfe0 2024-01-11T05:02:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.320+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbd94024a50 con 0x7fbda00fbfe0 2024-01-11T05:02:39.325 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.321+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda00591e0 con 0x7fbda00fbfe0 2024-01-11T05:02:39.325 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.323+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbd9401f020 con 0x7fbda00fbfe0 2024-01-11T05:02:39.391 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.390+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fbda010ce00 con 0x7fbda00fbfe0 2024-01-11T05:02:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.395+0000 7fbd87fff700 1 -- 172.21.15.67:0/3355143462 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbd94016070 con 0x7fbda00fbfe0 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 msgr2=0x7fbda010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 --2- 172.21.15.67:0/3355143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda010ad30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbd94000c00 tx=0x7fbd94006ee0).stop 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 shutdown_connections 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 --2- 172.21.15.67:0/3355143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbda00fbfe0 0x7fbda010ad30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 >> 172.21.15.67:0/3355143462 conn(0x7fbda00f7790 msgr2=0x7fbda00f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 shutdown_connections 2024-01-11T05:02:39.398 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:39.396+0000 7fbda5d20700 1 -- 172.21.15.67:0/3355143462 wait complete. 2024-01-11T05:02:39.782 INFO:teuthology.orchestra.run.smithi067.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-11T05:02:39.782 INFO:teuthology.orchestra.run.smithi067.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-11T05:02:39.782 INFO:teuthology.orchestra.run.smithi067.stderr:Creating mgr... 2024-01-11T05:02:39.783 INFO:teuthology.orchestra.run.smithi067.stderr:Verifying port 9283 ... 2024-01-11T05:02:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Failed to reset failed state of unit ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mgr.smithi067.srbzpc.service: Unit ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mgr.smithi067.srbzpc.service not loaded. 2024-01-11T05:02:39.919 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: Created symlink /etc/systemd/system/ceph-89773ea0-b03e-11ee-95ab-87774f69a715.target.wants/ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mgr.smithi067.srbzpc.service → /etc/systemd/system/ceph-89773ea0-b03e-11ee-95ab-87774f69a715@.service. 2024-01-11T05:02:40.433 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-01-11T05:02:40.442 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-01-11T05:02:40.442 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-01-11T05:02:40.442 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to enable service . firewalld.service is not available 2024-01-11T05:02:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-01-11T05:02:40.458 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-01-11T05:02:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-01-11T05:02:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-11T05:02:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr to start... 2024-01-11T05:02:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr... 2024-01-11T05:02:40.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:40 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:39.397341+0000 mon.smithi067 (mon.0) 6 : audit [INF] from='client.? 172.21.15.67:0/3355143462' entity='client.admin' 2024-01-11T05:02:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.835+0000 7f4c3c133700 1 Processor -- start 2024-01-11T05:02:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.835+0000 7f4c3c133700 1 -- start start 2024-01-11T05:02:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.835+0000 7f4c3c133700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c340fec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.835+0000 7f4c3c133700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34027910 con 0x7f4c340274f0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c39ecf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c340fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c39ecf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c340fec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44022/0 (socket says 172.21.15.67:44022) 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c39ecf700 1 -- 172.21.15.67:0/3726896649 learned_addr learned my addr 172.21.15.67:0/3726896649 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c39ecf700 1 -- 172.21.15.67:0/3726896649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340ff120 con 0x7f4c340274f0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c39ecf700 1 --2- 172.21.15.67:0/3726896649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c340fec30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f4c3000b5f0 tx=0x7f4c30006ee0).ready entity=mon.0 client_cookie=a0dd003a170270f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c38ecd700 1 -- 172.21.15.67:0/3726896649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c3000d600 con 0x7f4c340274f0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c38ecd700 1 -- 172.21.15.67:0/3726896649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4c3000dbc0 con 0x7f4c340274f0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.836+0000 7f4c38ecd700 1 -- 172.21.15.67:0/3726896649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c30011b90 con 0x7f4c340274f0 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 -- 172.21.15.67:0/3726896649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 msgr2=0x7f4c340fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 --2- 172.21.15.67:0/3726896649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c340fec30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f4c3000b5f0 tx=0x7f4c30006ee0).stop 2024-01-11T05:02:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 -- 172.21.15.67:0/3726896649 shutdown_connections 2024-01-11T05:02:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 --2- 172.21.15.67:0/3726896649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c340fec30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 -- 172.21.15.67:0/3726896649 >> 172.21.15.67:0/3726896649 conn(0x7f4c340f65f0 msgr2=0x7f4c340f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 -- 172.21.15.67:0/3726896649 shutdown_connections 2024-01-11T05:02:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.837+0000 7f4c3c133700 1 -- 172.21.15.67:0/3726896649 wait complete. 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.838+0000 7f4c3c133700 1 Processor -- start 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.838+0000 7f4c3c133700 1 -- start start 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c3c133700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c3410ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c3c133700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3410b220 con 0x7f4c340274f0 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c39ecf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c3410ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c39ecf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c3410ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44030/0 (socket says 172.21.15.67:44030) 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c39ecf700 1 -- 172.21.15.67:0/2074463634 learned_addr learned my addr 172.21.15.67:0/2074463634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c39ecf700 1 -- 172.21.15.67:0/2074463634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3000b200 con 0x7f4c340274f0 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c39ecf700 1 --2- 172.21.15.67:0/2074463634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c3410ad30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4c30006800 tx=0x7f4c30006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c3000de50 con 0x7f4c340274f0 2024-01-11T05:02:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4c3001b870 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3410b420 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.839+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c3001a7a0 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.840+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3410bfd0 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.840+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f4c3000f070 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.840+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4c30024d70 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.841+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c340591e0 con 0x7f4c340274f0 2024-01-11T05:02:40.844 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.843+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4c3001f030 con 0x7f4c340274f0 2024-01-11T05:02:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.914+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4c34100000 con 0x7f4c340274f0 2024-01-11T05:02:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.914+0000 7f4c227fc700 1 -- 172.21.15.67:0/2074463634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f4c30033090 con 0x7f4c340274f0 2024-01-11T05:02:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:02:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsid": "89773ea0-b03e-11ee-95ab-87774f69a715", 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "health": { 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "checks": {}, 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mutes": [] 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum": [ 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 0 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-11T05:02:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "smithi067" 2024-01-11T05:02:40.925 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:40.925 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-11T05:02:40.925 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "monmap": { 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osdmap": { 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-11T05:02:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgmap": { 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsmap": { 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-11T05:02:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": false, 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modules": [ 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "iostat", 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "nfs", 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "restful" 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "servicemap": { 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:40.928 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modified": "2024-01-11T05:02:35.373234+0000", 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 msgr2=0x7f4c3410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 --2- 172.21.15.67:0/2074463634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c3410ad30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4c30006800 tx=0x7f4c30006ee0).stop 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 shutdown_connections 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 --2- 172.21.15.67:0/2074463634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340274f0 0x7f4c3410ad30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 >> 172.21.15.67:0/2074463634 conn(0x7f4c340f65f0 msgr2=0x7f4c340f8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 shutdown_connections 2024-01-11T05:02:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:40.915+0000 7f4c3c133700 1 -- 172.21.15.67:0/2074463634 wait complete. 2024-01-11T05:02:41.256 INFO:teuthology.orchestra.run.smithi067.stderr:mgr not available, waiting (1/15)... 2024-01-11T05:02:41.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:40.916157+0000 mon.smithi067 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.67:0/2074463634' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-11T05:02:45.107 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fed1bb700 1 Processor -- start 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fed1bb700 1 -- start start 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fed1bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe80fec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fed1bb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80ff110 con 0x7f8fe8027630 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fe6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe80fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fe6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe80fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59638/0 (socket says 172.21.15.67:59638) 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fe6d9d700 1 -- 172.21.15.67:0/4041732119 learned_addr learned my addr 172.21.15.67:0/4041732119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fe6d9d700 1 -- 172.21.15.67:0/4041732119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80ff250 con 0x7f8fe8027630 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.091+0000 7f8fe6d9d700 1 --2- 172.21.15.67:0/4041732119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe80fec20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8fd800b5f0 tx=0x7f8fd8006ee0).ready entity=mon.0 client_cookie=d7072e5f54d1b329 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fe5d9b700 1 -- 172.21.15.67:0/4041732119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd800d600 con 0x7f8fe8027630 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fe5d9b700 1 -- 172.21.15.67:0/4041732119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8fd800dbc0 con 0x7f8fe8027630 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fe5d9b700 1 -- 172.21.15.67:0/4041732119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd8011b90 con 0x7f8fe8027630 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 -- 172.21.15.67:0/4041732119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 msgr2=0x7f8fe80fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 --2- 172.21.15.67:0/4041732119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe80fec20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8fd800b5f0 tx=0x7f8fd8006ee0).stop 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 -- 172.21.15.67:0/4041732119 shutdown_connections 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 --2- 172.21.15.67:0/4041732119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe80fec20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 -- 172.21.15.67:0/4041732119 >> 172.21.15.67:0/4041732119 conn(0x7f8fe80f6680 msgr2=0x7f8fe80f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 -- 172.21.15.67:0/4041732119 shutdown_connections 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.092+0000 7f8fed1bb700 1 -- 172.21.15.67:0/4041732119 wait complete. 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fed1bb700 1 Processor -- start 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fed1bb700 1 -- start start 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fed1bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe810ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fed1bb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe810b220 con 0x7f8fe8027630 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fe6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe810ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fe6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe810ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59654/0 (socket says 172.21.15.67:59654) 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fe6d9d700 1 -- 172.21.15.67:0/1436515266 learned_addr learned my addr 172.21.15.67:0/1436515266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.093+0000 7f8fe6d9d700 1 -- 172.21.15.67:0/1436515266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd800b200 con 0x7f8fe8027630 2024-01-11T05:02:45.109 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fe6d9d700 1 --2- 172.21.15.67:0/1436515266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe810ad30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f8fd8006800 tx=0x7f8fd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd800de50 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe810b420 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8fd801b870 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd801a7a0 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe810c030 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f8fd800f070 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.094+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8fd8024c40 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.095+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe80591e0 con 0x7f8fe8027630 2024-01-11T05:02:45.110 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.097+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8fd801f030 con 0x7f8fe8027630 2024-01-11T05:02:45.175 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.168+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8fe80fff90 con 0x7f8fe8027630 2024-01-11T05:02:45.175 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.169+0000 7f8fcf7fe700 1 -- 172.21.15.67:0/1436515266 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f8fd8033090 con 0x7f8fe8027630 2024-01-11T05:02:45.175 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.175 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:02:45.175 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsid": "89773ea0-b03e-11ee-95ab-87774f69a715", 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "health": { 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "checks": {}, 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mutes": [] 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum": [ 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 0 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "smithi067" 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "monmap": { 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osdmap": { 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgmap": { 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsmap": { 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": false, 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modules": [ 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "iostat", 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "nfs", 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "restful" 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "servicemap": { 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modified": "2024-01-11T05:02:35.373234+0000", 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.169+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 msgr2=0x7f8fe810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.169+0000 7f8fed1bb700 1 --2- 172.21.15.67:0/1436515266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe810ad30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f8fd8006800 tx=0x7f8fd8006ee0).stop 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.170+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 shutdown_connections 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.170+0000 7f8fed1bb700 1 --2- 172.21.15.67:0/1436515266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fe8027630 0x7f8fe810ad30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.170+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 >> 172.21.15.67:0/1436515266 conn(0x7f8fe80f6680 msgr2=0x7f8fe80f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.170+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 shutdown_connections 2024-01-11T05:02:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:45.170+0000 7f8fed1bb700 1 -- 172.21.15.67:0/1436515266 wait complete. 2024-01-11T05:02:45.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.169674+0000 mon.smithi067 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.67:0/1436515266' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:45.811753+0000 mon.smithi067 (mon.0) 9 : cluster [INF] Activating manager daemon smithi067.srbzpc 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:45.814410+0000 mon.smithi067 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi067.srbzpc(active, starting, since 0.00275373s) 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.815385+0000 mon.smithi067 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.815566+0000 mon.smithi067 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.815678+0000 mon.smithi067 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.815815+0000 mon.smithi067 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.816208+0000 mon.smithi067 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mgr metadata", "who": "smithi067.srbzpc", "id": "smithi067.srbzpc"}]: dispatch 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:45.818016+0000 mon.smithi067 (mon.0) 16 : cluster [INF] Manager daemon smithi067.srbzpc is now available 2024-01-11T05:02:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.824447+0000 mon.smithi067 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' 2024-01-11T05:02:46.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.830789+0000 mon.smithi067 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:02:46.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.832040+0000 mon.smithi067 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:02:46.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.837431+0000 mon.smithi067 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' 2024-01-11T05:02:46.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:45.839583+0000 mon.smithi067 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.67:0/874752777' entity='mgr.smithi067.srbzpc' 2024-01-11T05:02:46.605 INFO:teuthology.orchestra.run.smithi067.stderr:mgr not available, waiting (2/15)... 2024-01-11T05:02:48.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:46.817417+0000 mon.smithi067 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi067.srbzpc(active, since 1.00576s) 2024-01-11T05:02:49.865 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:48.818270+0000 mon.smithi067 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi067.srbzpc(active, since 3s) 2024-01-11T05:02:50.767 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.590+0000 7f34eb561700 1 Processor -- start 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.590+0000 7f34eb561700 1 -- start start 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.590+0000 7f34eb561700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.590+0000 7f34eb561700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e40fc8a0 con 0x7f34e406ad60 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34ea55f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34ea55f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59686/0 (socket says 172.21.15.67:59686) 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34ea55f700 1 -- 172.21.15.67:0/500015793 learned_addr learned my addr 172.21.15.67:0/500015793 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34ea55f700 1 -- 172.21.15.67:0/500015793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e41013a0 con 0x7f34e406ad60 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34ea55f700 1 --2- 172.21.15.67:0/500015793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e4100eb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f34e000b5f0 tx=0x7f34e0006ee0).ready entity=mon.0 client_cookie=40c79f026013459f server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34e955d700 1 -- 172.21.15.67:0/500015793 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e000d600 con 0x7f34e406ad60 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34e955d700 1 -- 172.21.15.67:0/500015793 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f34e000dbc0 con 0x7f34e406ad60 2024-01-11T05:02:50.768 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.591+0000 7f34e955d700 1 -- 172.21.15.67:0/500015793 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e0011b90 con 0x7f34e406ad60 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 -- 172.21.15.67:0/500015793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 msgr2=0x7f34e4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 --2- 172.21.15.67:0/500015793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e4100eb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f34e000b5f0 tx=0x7f34e0006ee0).stop 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 -- 172.21.15.67:0/500015793 shutdown_connections 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 --2- 172.21.15.67:0/500015793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e4100eb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 -- 172.21.15.67:0/500015793 >> 172.21.15.67:0/500015793 conn(0x7f34e40f64d0 msgr2=0x7f34e40f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 -- 172.21.15.67:0/500015793 shutdown_connections 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 -- 172.21.15.67:0/500015793 wait complete. 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.592+0000 7f34eb561700 1 Processor -- start 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34eb561700 1 -- start start 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34eb561700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e410cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34eb561700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e410d2a0 con 0x7f34e406ad60 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34ea55f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e410cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:50.769 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34ea55f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e410cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59688/0 (socket says 172.21.15.67:59688) 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34ea55f700 1 -- 172.21.15.67:0/1704877667 learned_addr learned my addr 172.21.15.67:0/1704877667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34ea55f700 1 -- 172.21.15.67:0/1704877667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e000b200 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34ea55f700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e410cdb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f34e0006800 tx=0x7f34e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e000d9c0 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f34e001b870 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e410d4a0 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e001a740 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.593+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e410e0e0 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.594+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f34e002b070 con 0x7f34e406ad60 2024-01-11T05:02:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.594+0000 7f34d2ffd700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f34cc030fa0 0x7f34cc033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.594+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f34e0042b80 con 0x7f34e406ad60 2024-01-11T05:02:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.594+0000 7f34e9d5e700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f34cc030fa0 0x7f34cc033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.594+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e410d930 con 0x7f34e406ad60 2024-01-11T05:02:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.595+0000 7f34e9d5e700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f34cc030fa0 0x7f34cc033460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f34d40096a0 tx=0x7f34d4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.599+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f34e0016020 con 0x7f34e406ad60 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.819+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f34e4101f80 con 0x7f34e406ad60 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.819+0000 7f34d2ffd700 1 -- 172.21.15.67:0/1704877667 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f34e0016020 con 0x7f34e406ad60 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsid": "89773ea0-b03e-11ee-95ab-87774f69a715", 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "health": { 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "checks": {}, 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mutes": [] 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.827 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum": [ 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 0 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "smithi067" 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "monmap": { 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osdmap": { 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgmap": { 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-11T05:02:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "fsmap": { 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": true, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modules": [ 2024-01-11T05:02:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "iostat", 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "nfs", 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "restful" 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ], 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "servicemap": { 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "modified": "2024-01-11T05:02:35.373234+0000", 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "services": {} 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: }, 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f34cc030fa0 msgr2=0x7f34cc033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f34cc030fa0 0x7f34cc033460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f34d40096a0 tx=0x7f34d4006b40).stop 2024-01-11T05:02:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 msgr2=0x7f34e410cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e410cdb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f34e0006800 tx=0x7f34e0006ee0).stop 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 shutdown_connections 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f34cc030fa0 0x7f34cc033460 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 --2- 172.21.15.67:0/1704877667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e406ad60 0x7f34e410cdb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 >> 172.21.15.67:0/1704877667 conn(0x7f34e40f64d0 msgr2=0x7f34e40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.821+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 shutdown_connections 2024-01-11T05:02:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:50.822+0000 7f34eb561700 1 -- 172.21.15.67:0/1704877667 wait complete. 2024-01-11T05:02:51.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:50.820549+0000 mon.smithi067 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.67:0/1704877667' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-11T05:02:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:mgr is available 2024-01-11T05:02:52.481 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.480+0000 7fb073d2f700 1 Processor -- start 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.480+0000 7fb073d2f700 1 -- start start 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.480+0000 7fb073d2f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.480+0000 7fb073d2f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0ff0d0 con 0x7fb06c027670 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.480+0000 7fb071acb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb071acb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35240/0 (socket says 172.21.15.67:35240) 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb071acb700 1 -- 172.21.15.67:0/704121874 learned_addr learned my addr 172.21.15.67:0/704121874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb071acb700 1 -- 172.21.15.67:0/704121874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c0ff210 con 0x7fb06c027670 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb071acb700 1 --2- 172.21.15.67:0/704121874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0febe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb05c00b5f0 tx=0x7fb05c006ee0).ready entity=mon.0 client_cookie=4d32c81d4d1646d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb070ac9700 1 -- 172.21.15.67:0/704121874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb05c00d600 con 0x7fb06c027670 2024-01-11T05:02:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb070ac9700 1 -- 172.21.15.67:0/704121874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb05c00dbc0 con 0x7fb06c027670 2024-01-11T05:02:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.481+0000 7fb070ac9700 1 -- 172.21.15.67:0/704121874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb05c011b90 con 0x7fb06c027670 2024-01-11T05:02:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 -- 172.21.15.67:0/704121874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 msgr2=0x7fb06c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 --2- 172.21.15.67:0/704121874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0febe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb05c00b5f0 tx=0x7fb05c006ee0).stop 2024-01-11T05:02:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 -- 172.21.15.67:0/704121874 shutdown_connections 2024-01-11T05:02:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 --2- 172.21.15.67:0/704121874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0febe0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 -- 172.21.15.67:0/704121874 >> 172.21.15.67:0/704121874 conn(0x7fb06c0f6680 msgr2=0x7fb06c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 -- 172.21.15.67:0/704121874 shutdown_connections 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 -- 172.21.15.67:0/704121874 wait complete. 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 Processor -- start 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.482+0000 7fb073d2f700 1 -- start start 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb073d2f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb073d2f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0faae0 con 0x7fb06c027670 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb071acb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb071acb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35250/0 (socket says 172.21.15.67:35250) 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb071acb700 1 -- 172.21.15.67:0/4106605857 learned_addr learned my addr 172.21.15.67:0/4106605857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb071acb700 1 -- 172.21.15.67:0/4106605857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c00b200 con 0x7fb06c027670 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb071acb700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0fc660 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fb05c000c00 tx=0x7fb05c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb05c01b410 con 0x7fb06c027670 2024-01-11T05:02:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fb05c01b9d0 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c0fac80 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb05c01a620 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.483+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c0fb110 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.484+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7fb05c02b070 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.484+0000 7fb05a7fc700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fb054030fa0 0x7fb054033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.484+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb05c042ac0 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.484+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c045bb0 con 0x7fb06c027670 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.485+0000 7fb0712ca700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fb054030fa0 0x7fb054033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.485+0000 7fb0712ca700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fb054030fa0 0x7fb054033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb0680096a0 tx=0x7fb068006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:52.489 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.489+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb05c016020 con 0x7fb06c027670 2024-01-11T05:02:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.648+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb06c0fff10 con 0x7fb06c027670 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.649+0000 7fb05a7fc700 1 -- 172.21.15.67:0/4106605857 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7fb05c01ae00 con 0x7fb06c027670 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [global] 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: fsid = 89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: log_to_journald = false 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [mgr] 2024-01-11T05:02:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: [osd] 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fb054030fa0 msgr2=0x7fb054033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fb054030fa0 0x7fb054033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb0680096a0 tx=0x7fb068006b40).stop 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 msgr2=0x7fb06c0fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0fc660 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fb05c000c00 tx=0x7fb05c006ee0).stop 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 shutdown_connections 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fb054030fa0 0x7fb054033460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:52.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 --2- 172.21.15.67:0/4106605857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb06c027670 0x7fb06c0fc660 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:52.655 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.651+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 >> 172.21.15.67:0/4106605857 conn(0x7fb06c0f6680 msgr2=0x7fb06c0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:52.655 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.652+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 shutdown_connections 2024-01-11T05:02:52.655 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:52.652+0000 7fb073d2f700 1 -- 172.21.15.67:0/4106605857 wait complete. 2024-01-11T05:02:52.965 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:52.649764+0000 mon.smithi067 (mon.0) 25 : audit [INF] from='client.? 172.21.15.67:0/4106605857' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-11T05:02:53.013 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling cephadm module... 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75cd365700 1 Processor -- start 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75cd365700 1 -- start start 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75cd365700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c80fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75cd365700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80fb6b0 con 0x7f75c80fada0 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75c6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c80fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75c6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c80fb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35254/0 (socket says 172.21.15.67:35254) 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75c6ffd700 1 -- 172.21.15.67:0/445674309 learned_addr learned my addr 172.21.15.67:0/445674309 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75c6ffd700 1 -- 172.21.15.67:0/445674309 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80fb7f0 con 0x7f75c80fada0 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.327+0000 7f75c6ffd700 1 --2- 172.21.15.67:0/445674309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c80fb1c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f75b800b870 tx=0x7f75b8006f70).ready entity=mon.0 client_cookie=678bdac5ff6241b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75c5ffb700 1 -- 172.21.15.67:0/445674309 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b800da20 con 0x7f75c80fada0 2024-01-11T05:02:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75c5ffb700 1 -- 172.21.15.67:0/445674309 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f75b8010470 con 0x7f75c80fada0 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75c5ffb700 1 -- 172.21.15.67:0/445674309 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b800f5b0 con 0x7f75c80fada0 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 -- 172.21.15.67:0/445674309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 msgr2=0x7f75c80fb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 --2- 172.21.15.67:0/445674309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c80fb1c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f75b800b870 tx=0x7f75b8006f70).stop 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 -- 172.21.15.67:0/445674309 shutdown_connections 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 --2- 172.21.15.67:0/445674309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c80fb1c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 -- 172.21.15.67:0/445674309 >> 172.21.15.67:0/445674309 conn(0x7f75c80f6500 msgr2=0x7f75c80f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 -- 172.21.15.67:0/445674309 shutdown_connections 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.328+0000 7f75cd365700 1 -- 172.21.15.67:0/445674309 wait complete. 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75cd365700 1 Processor -- start 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75cd365700 1 -- start start 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75cd365700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c8108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75cd365700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8108e60 con 0x7f75c80fada0 2024-01-11T05:02:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75c6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c8108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75c6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c8108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35270/0 (socket says 172.21.15.67:35270) 2024-01-11T05:02:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.329+0000 7f75c6ffd700 1 -- 172.21.15.67:0/574493294 learned_addr learned my addr 172.21.15.67:0/574493294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75c6ffd700 1 -- 172.21.15.67:0/574493294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b800b480 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75c6ffd700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c8108970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f75b8010d50 tx=0x7f75b8006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b800f3d0 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f75b8006c10 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b8019bb0 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c8109060 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.330+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81094f0 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.331+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f75b8016030 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.331+0000 7f75af7fe700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f75b4030fa0 0x7f75b4033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.331+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f75b8045870 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.331+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c8045bb0 con 0x7f75c80fada0 2024-01-11T05:02:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.331+0000 7f75c67fc700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f75b4030fa0 0x7f75b4033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.331+0000 7f75c67fc700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f75b4030fa0 0x7f75b4033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f75bc0096a0 tx=0x7f75bc006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:53.335 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.335+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f75b8014070 con 0x7f75c80fada0 2024-01-11T05:02:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.537+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f75c80fca50 con 0x7f75c80fada0 2024-01-11T05:02:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.706+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f75b802da00 con 0x7f75c80fada0 2024-01-11T05:02:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.707+0000 7f75af7fe700 1 -- 172.21.15.67:0/574493294 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f75b802c5c0 con 0x7f75c80fada0 2024-01-11T05:02:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f75b4030fa0 msgr2=0x7f75b4033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f75b4030fa0 0x7f75b4033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f75bc0096a0 tx=0x7f75bc006b40).stop 2024-01-11T05:02:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 msgr2=0x7f75c8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c8108970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f75b8010d50 tx=0x7f75b8006f70).stop 2024-01-11T05:02:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 shutdown_connections 2024-01-11T05:02:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7f75b4030fa0 0x7f75b4033460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 --2- 172.21.15.67:0/574493294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c80fada0 0x7f75c8108970 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 >> 172.21.15.67:0/574493294 conn(0x7f75c80f6500 msgr2=0x7f75c80ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 shutdown_connections 2024-01-11T05:02:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:53.709+0000 7f75cd365700 1 -- 172.21.15.67:0/574493294 wait complete. 2024-01-11T05:02:53.974 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:53.538237+0000 mon.smithi067 (mon.0) 26 : audit [INF] from='client.? 172.21.15.67:0/574493294' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-11T05:02:54.440 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82759e700 1 Processor -- start 2024-01-11T05:02:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82759e700 1 -- start start 2024-01-11T05:02:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa8280fb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82759e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280fb680 con 0x7fa8280fad70 2024-01-11T05:02:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa8280fb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa8280fb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35304/0 (socket says 172.21.15.67:35304) 2024-01-11T05:02:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82659c700 1 -- 172.21.15.67:0/2084999205 learned_addr learned my addr 172.21.15.67:0/2084999205 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.440+0000 7fa82659c700 1 -- 172.21.15.67:0/2084999205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280fb7c0 con 0x7fa8280fad70 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.441+0000 7fa82659c700 1 --2- 172.21.15.67:0/2084999205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa8280fb190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fa81800b5f0 tx=0x7fa818006ee0).ready entity=mon.0 client_cookie=1f28e764f03bf3cf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.441+0000 7fa82559a700 1 -- 172.21.15.67:0/2084999205 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa81800d600 con 0x7fa8280fad70 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.441+0000 7fa82559a700 1 -- 172.21.15.67:0/2084999205 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa81800dbc0 con 0x7fa8280fad70 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.441+0000 7fa82559a700 1 -- 172.21.15.67:0/2084999205 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa818011b90 con 0x7fa8280fad70 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 -- 172.21.15.67:0/2084999205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 msgr2=0x7fa8280fb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 --2- 172.21.15.67:0/2084999205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa8280fb190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fa81800b5f0 tx=0x7fa818006ee0).stop 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 -- 172.21.15.67:0/2084999205 shutdown_connections 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 --2- 172.21.15.67:0/2084999205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa8280fb190 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 -- 172.21.15.67:0/2084999205 >> 172.21.15.67:0/2084999205 conn(0x7fa8280f64d0 msgr2=0x7fa8280f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 -- 172.21.15.67:0/2084999205 shutdown_connections 2024-01-11T05:02:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 -- 172.21.15.67:0/2084999205 wait complete. 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 Processor -- start 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.442+0000 7fa82759e700 1 -- start start 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa828109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82759e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828109ef0 con 0x7fa8280fad70 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa828109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa828109a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35308/0 (socket says 172.21.15.67:35308) 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82659c700 1 -- 172.21.15.67:0/3300212512 learned_addr learned my addr 172.21.15.67:0/3300212512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82659c700 1 -- 172.21.15.67:0/3300212512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81800b200 con 0x7fa8280fad70 2024-01-11T05:02:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82659c700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa828109a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa81800e040 tx=0x7fa818006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa81800de00 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa81801b870 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa81801a6c0 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82810a0f0 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.443+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82810a580 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.444+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa81800f070 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.444+0000 7fa816ffd700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 0x7fa80c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.444+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa818042a20 con 0x7fa8280fad70 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.444+0000 7fa825d9b700 1 -- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 msgr2=0x7fa80c033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.444+0000 7fa825d9b700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 0x7fa80c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-11T05:02:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.445+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828045bb0 con 0x7fa8280fad70 2024-01-11T05:02:54.449 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.449+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa818016030 con 0x7fa8280fad70 2024-01-11T05:02:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.645+0000 7fa825d9b700 1 -- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 msgr2=0x7fa80c033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.645+0000 7fa825d9b700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 0x7fa80c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-11T05:02:54.658 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.658+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa828103a60 con 0x7fa8280fad70 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.658+0000 7fa816ffd700 1 -- 172.21.15.67:0/3300212512 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa81801b410 con 0x7fa8280fad70 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": true, 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "active_name": "smithi067.srbzpc", 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-11T05:02:54.659 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 msgr2=0x7fa80c033460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 0x7fa80c033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 msgr2=0x7fa828109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa828109a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa81800e040 tx=0x7fa818006ee0).stop 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 shutdown_connections 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fa80c030fa0 0x7fa80c033460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:02:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 --2- 172.21.15.67:0/3300212512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8280fad70 0x7fa828109a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 >> 172.21.15.67:0/3300212512 conn(0x7fa8280f64d0 msgr2=0x7fa8280f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 shutdown_connections 2024-01-11T05:02:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:54.660+0000 7fa82759e700 1 -- 172.21.15.67:0/3300212512 wait complete. 2024-01-11T05:02:54.957 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:54 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:53.708290+0000 mon.smithi067 (mon.0) 27 : audit [INF] from='client.? 172.21.15.67:0/574493294' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-11T05:02:54.958 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:54 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:53.708335+0000 mon.smithi067 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi067.srbzpc(active, since 7s) 2024-01-11T05:02:54.958 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:54 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:54.659246+0000 mon.smithi067 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.67:0/3300212512' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-11T05:02:55.017 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for the mgr to restart... 2024-01-11T05:02:55.017 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr epoch 5... 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.422+0000 7fdb81038700 1 Processor -- start 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.422+0000 7fdb81038700 1 -- start start 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.422+0000 7fdb81038700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c0fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.422+0000 7fdb81038700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c0fb7e0 con 0x7fdb7c0faed0 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.422+0000 7fdb7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c0fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c0fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35322/0 (socket says 172.21.15.67:35322) 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb7ad9d700 1 -- 172.21.15.67:0/1938708039 learned_addr learned my addr 172.21.15.67:0/1938708039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb7ad9d700 1 -- 172.21.15.67:0/1938708039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c0fb920 con 0x7fdb7c0faed0 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb7ad9d700 1 --2- 172.21.15.67:0/1938708039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c0fb2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fdb6c00b5f0 tx=0x7fdb6c006ee0).ready entity=mon.0 client_cookie=5bf09c1bb370d900 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb79d9b700 1 -- 172.21.15.67:0/1938708039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c00d600 con 0x7fdb7c0faed0 2024-01-11T05:02:55.425 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb79d9b700 1 -- 172.21.15.67:0/1938708039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fdb6c00dbc0 con 0x7fdb7c0faed0 2024-01-11T05:02:55.425 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.423+0000 7fdb79d9b700 1 -- 172.21.15.67:0/1938708039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c011b90 con 0x7fdb7c0faed0 2024-01-11T05:02:55.427 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 -- 172.21.15.67:0/1938708039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 msgr2=0x7fdb7c0fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 --2- 172.21.15.67:0/1938708039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c0fb2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fdb6c00b5f0 tx=0x7fdb6c006ee0).stop 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 -- 172.21.15.67:0/1938708039 shutdown_connections 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 --2- 172.21.15.67:0/1938708039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c0fb2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 -- 172.21.15.67:0/1938708039 >> 172.21.15.67:0/1938708039 conn(0x7fdb7c0f6680 msgr2=0x7fdb7c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 -- 172.21.15.67:0/1938708039 shutdown_connections 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.424+0000 7fdb81038700 1 -- 172.21.15.67:0/1938708039 wait complete. 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb81038700 1 Processor -- start 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb81038700 1 -- start start 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb81038700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb81038700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c10d3e0 con 0x7fdb7c0faed0 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:02:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c10cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35332/0 (socket says 172.21.15.67:35332) 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb7ad9d700 1 -- 172.21.15.67:0/2627000546 learned_addr learned my addr 172.21.15.67:0/2627000546 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.425+0000 7fdb7ad9d700 1 -- 172.21.15.67:0/2627000546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c00b200 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.426+0000 7fdb7ad9d700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c10cef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fdb6c000c00 tx=0x7fdb6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.426+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c01b410 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.426+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fdb6c01b9d0 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.426+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7c10d5e0 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.426+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c01a620 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.426+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c10da70 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.427+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fdb6c02b070 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.427+0000 7fdb637fe700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.427+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 --> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb7c0591e0 con 0x7fdb68030fa0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.427+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdb6c042ba0 con 0x7fdb7c0faed0 2024-01-11T05:02:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.427+0000 7fdb7a59c700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 msgr2=0x7fdb68033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:55.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.427+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-11T05:02:55.628 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.628+0000 7fdb7a59c700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 msgr2=0x7fdb68033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:55.628 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:55.628+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-11T05:02:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:56.029+0000 7fdb7a59c700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 msgr2=0x7fdb68033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:56.029+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-11T05:02:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:56.830+0000 7fdb7a59c700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 msgr2=0x7fdb68033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:56.830+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-11T05:02:58.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:58.431+0000 7fdb7a59c700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 msgr2=0x7fdb68033460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/653150909 2024-01-11T05:02:58.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:58.431+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-11T05:02:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:59.994+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 6) v1 ==== 36945+0+0 (secure 0 0 0) 0x7fdb6c00f070 con 0x7fdb7c0faed0 2024-01-11T05:02:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:59.994+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 msgr2=0x7fdb68033460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T05:02:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:02:59.994+0000 7fdb637fe700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:03:00.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:59.055790+0000 mon.smithi067 (mon.0) 30 : cluster [INF] Active manager daemon smithi067.srbzpc restarted 2024-01-11T05:03:00.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:59.056216+0000 mon.smithi067 (mon.0) 31 : cluster [INF] Activating manager daemon smithi067.srbzpc 2024-01-11T05:03:00.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:02:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:59.059015+0000 mon.smithi067 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-11T05:03:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:00.996+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7fdb6c01bb40 con 0x7fdb7c0faed0 2024-01-11T05:03:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:00.996+0000 7fdb637fe700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fdb68034850 0x7fdb68036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:00.996+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb7c0591e0 con 0x7fdb68034850 2024-01-11T05:03:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:00.997+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fdb68034850 0x7fdb68036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:00.997+0000 7fdb7a59c700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fdb68034850 0x7fdb68036c40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdb70006fd0 tx=0x7fdb70006cf0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:00.998+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fdb7c0591e0 con 0x7fdb68034850 2024-01-11T05:03:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.007+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fdb7c045bb0 con 0x7fdb68034850 2024-01-11T05:03:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.007+0000 7fdb637fe700 1 -- 172.21.15.67:0/2627000546 <== mgr.14116 v2:172.21.15.67:6800/3176394990 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fdb7c045bb0 con 0x7fdb68034850 2024-01-11T05:03:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:03:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-11T05:03:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "initialized": true 2024-01-11T05:03:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.007+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fdb68034850 msgr2=0x7fdb68036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.007+0000 7fdb81038700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fdb68034850 0x7fdb68036c40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdb70006fd0 tx=0x7fdb70006cf0).stop 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.007+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 msgr2=0x7fdb7c10cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.007+0000 7fdb81038700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c10cef0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fdb6c000c00 tx=0x7fdb6c006ee0).stop 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 shutdown_connections 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fdb68034850 0x7fdb68036c40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c0faed0 0x7fdb7c10cef0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 --2- 172.21.15.67:0/2627000546 >> [v2:172.21.15.67:6800/653150909,v1:172.21.15.67:6801/653150909] conn(0x7fdb68030fa0 0x7fdb68033460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 >> 172.21.15.67:0/2627000546 conn(0x7fdb7c0f6680 msgr2=0x7fdb7c0271d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 shutdown_connections 2024-01-11T05:03:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.008+0000 7fdb81038700 1 -- 172.21.15.67:0/2627000546 wait complete. 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:02:59.996472+0000 mon.smithi067 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi067.srbzpc(active, starting, since 0.940322s) 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:59.998102+0000 mon.smithi067 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:02:59.999052+0000 mon.smithi067 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mgr metadata", "who": "smithi067.srbzpc", "id": "smithi067.srbzpc"}]: dispatch 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.000211+0000 mon.smithi067 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.000424+0000 mon.smithi067 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.000593+0000 mon.smithi067 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:00.003172+0000 mon.smithi067 (mon.0) 39 : cluster [INF] Manager daemon smithi067.srbzpc is now available 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.019780+0000 mon.smithi067 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.023163+0000 mon.smithi067 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.026320+0000 mon.smithi067 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:01.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.127485+0000 mon.smithi067 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:01.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.128290+0000 mon.smithi067 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:01.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.129936+0000 mon.smithi067 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:01.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.135088+0000 mon.smithi067 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:01.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.142856+0000 mon.smithi067 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:03:01.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.144098+0000 mon.smithi067 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:03:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:mgr epoch 5 is available 2024-01-11T05:03:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:Setting orchestrator backend to cephadm... 2024-01-11T05:03:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.725+0000 7f4a69b5e700 1 Processor -- start 2024-01-11T05:03:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.725+0000 7f4a69b5e700 1 -- start start 2024-01-11T05:03:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.725+0000 7f4a69b5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a640fdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.725+0000 7f4a69b5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a640fdf90 con 0x7f4a640fd680 2024-01-11T05:03:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a640fdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a640fdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35402/0 (socket says 172.21.15.67:35402) 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a637fe700 1 -- 172.21.15.67:0/709830091 learned_addr learned my addr 172.21.15.67:0/709830091 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a637fe700 1 -- 172.21.15.67:0/709830091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a640fe0d0 con 0x7f4a640fd680 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a637fe700 1 --2- 172.21.15.67:0/709830091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a640fdaa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4a5400b5f0 tx=0x7f4a54006ee0).ready entity=mon.0 client_cookie=a66c11dbe863757e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a627fc700 1 -- 172.21.15.67:0/709830091 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5400d600 con 0x7f4a640fd680 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a627fc700 1 -- 172.21.15.67:0/709830091 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4a5400dbc0 con 0x7f4a640fd680 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.726+0000 7f4a627fc700 1 -- 172.21.15.67:0/709830091 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a54011b90 con 0x7f4a640fd680 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 -- 172.21.15.67:0/709830091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 msgr2=0x7f4a640fdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 --2- 172.21.15.67:0/709830091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a640fdaa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f4a5400b5f0 tx=0x7f4a54006ee0).stop 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 -- 172.21.15.67:0/709830091 shutdown_connections 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 --2- 172.21.15.67:0/709830091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a640fdaa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 -- 172.21.15.67:0/709830091 >> 172.21.15.67:0/709830091 conn(0x7f4a640f5e20 msgr2=0x7f4a640f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 -- 172.21.15.67:0/709830091 shutdown_connections 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 -- 172.21.15.67:0/709830091 wait complete. 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.727+0000 7f4a69b5e700 1 Processor -- start 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a69b5e700 1 -- start start 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a69b5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a6410ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a69b5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6410b280 con 0x7f4a640fd680 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a6410ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a6410ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35404/0 (socket says 172.21.15.67:35404) 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a637fe700 1 -- 172.21.15.67:0/3894111247 learned_addr learned my addr 172.21.15.67:0/3894111247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a637fe700 1 -- 172.21.15.67:0/3894111247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5400b200 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.728+0000 7f4a637fe700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a6410ad90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4a54006800 tx=0x7f4a54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5400de50 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f4a5401b870 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6410b480 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5401a7a0 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6410c090 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f4a5400f070 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a5bfff700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f4a44030f00 0x7f4a440333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.729+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f4a54042b50 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.730+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a640591e0 con 0x7f4a640fd680 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.730+0000 7f4a62ffd700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f4a44030f00 0x7f4a440333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.731+0000 7f4a62ffd700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f4a44030f00 0x7f4a440333c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4a4c0096a0 tx=0x7f4a4c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:01.735 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.734+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4a54016030 con 0x7f4a640fd680 2024-01-11T05:03:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.907+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a640fab80 con 0x7f4a44030f00 2024-01-11T05:03:01.912 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.912+0000 7f4a5bfff700 1 -- 172.21.15.67:0/3894111247 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a640fab80 con 0x7f4a44030f00 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f4a44030f00 msgr2=0x7f4a440333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f4a44030f00 0x7f4a440333c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4a4c0096a0 tx=0x7f4a4c006b40).stop 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 msgr2=0x7f4a6410ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a6410ad90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4a54006800 tx=0x7f4a54006ee0).stop 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 shutdown_connections 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f4a44030f00 0x7f4a440333c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 --2- 172.21.15.67:0/3894111247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a640fd680 0x7f4a6410ad90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 >> 172.21.15.67:0/3894111247 conn(0x7f4a640f5e20 msgr2=0x7f4a640ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:01.916 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.914+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 shutdown_connections 2024-01-11T05:03:01.917 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:01.915+0000 7f4a69b5e700 1 -- 172.21.15.67:0/3894111247 wait complete. 2024-01-11T05:03:02.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:00.998766+0000 mgr.smithi067.srbzpc (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-11T05:03:02.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:00.999773+0000 mon.smithi067 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi067.srbzpc(active, since 1.94362s) 2024-01-11T05:03:02.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:01.008177+0000 mgr.smithi067.srbzpc (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-11T05:03:02.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:01.912958+0000 mon.smithi067 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:02.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:01.917044+0000 mon.smithi067 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.645+0000 7f52a682e700 1 Processor -- start 2024-01-11T05:03:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.645+0000 7f52a682e700 1 -- start start 2024-01-11T05:03:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.645+0000 7f52a682e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a00fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.645+0000 7f52a682e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00fcde0 con 0x7f52a00fc4d0 2024-01-11T05:03:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a00fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a00fc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48132/0 (socket says 172.21.15.67:48132) 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529ffff700 1 -- 172.21.15.67:0/3168140015 learned_addr learned my addr 172.21.15.67:0/3168140015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529ffff700 1 -- 172.21.15.67:0/3168140015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a00fcf20 con 0x7f52a00fc4d0 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529ffff700 1 --2- 172.21.15.67:0/3168140015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a00fc8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f529000b870 tx=0x7f5290006f70).ready entity=mon.0 client_cookie=50165c07978fc96a server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529effd700 1 -- 172.21.15.67:0/3168140015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529000da20 con 0x7f52a00fc4d0 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529effd700 1 -- 172.21.15.67:0/3168140015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5290010470 con 0x7f52a00fc4d0 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.646+0000 7f529effd700 1 -- 172.21.15.67:0/3168140015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529000f5b0 con 0x7f52a00fc4d0 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 -- 172.21.15.67:0/3168140015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 msgr2=0x7f52a00fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 --2- 172.21.15.67:0/3168140015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a00fc8f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f529000b870 tx=0x7f5290006f70).stop 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 -- 172.21.15.67:0/3168140015 shutdown_connections 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 --2- 172.21.15.67:0/3168140015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a00fc8f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 -- 172.21.15.67:0/3168140015 >> 172.21.15.67:0/3168140015 conn(0x7f52a00f5e20 msgr2=0x7f52a00f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 -- 172.21.15.67:0/3168140015 shutdown_connections 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 -- 172.21.15.67:0/3168140015 wait complete. 2024-01-11T05:03:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.647+0000 7f52a682e700 1 Processor -- start 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f52a682e700 1 -- start start 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f52a682e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a0108b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f52a682e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0109050 con 0x7f52a00fc4d0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f529ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a0108b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f529ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a0108b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48136/0 (socket says 172.21.15.67:48136) 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f529ffff700 1 -- 172.21.15.67:0/1558619020 learned_addr learned my addr 172.21.15.67:0/1558619020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f529ffff700 1 -- 172.21.15.67:0/1558619020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529000b480 con 0x7f52a00fc4d0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.648+0000 7f529ffff700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a0108b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5290010d50 tx=0x7f5290006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.649+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529000f3d0 con 0x7f52a00fc4d0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.649+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a0109250 con 0x7f52a00fc4d0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.649+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5290006c10 con 0x7f52a00fc4d0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.649+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5290019b80 con 0x7f52a00fc4d0 2024-01-11T05:03:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.649+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a01096e0 con 0x7f52a00fc4d0 2024-01-11T05:03:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.649+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f5290016030 con 0x7f52a00fc4d0 2024-01-11T05:03:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.650+0000 7f529cff9700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f528c030fa0 0x7f528c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.650+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a0045bb0 con 0x7f52a00fc4d0 2024-01-11T05:03:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.650+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f5290045520 con 0x7f52a00fc4d0 2024-01-11T05:03:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.650+0000 7f529f7fe700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f528c030fa0 0x7f528c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.650+0000 7f529f7fe700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f528c030fa0 0x7f528c033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f52940096a0 tx=0x7f5294006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:02.654 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.654+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5290021370 con 0x7f52a00fc4d0 2024-01-11T05:03:02.821 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.820+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f52a010a000 con 0x7f528c030fa0 2024-01-11T05:03:02.821 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.821+0000 7f529cff9700 1 -- 172.21.15.67:0/1558619020 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f52a010a000 con 0x7f528c030fa0 2024-01-11T05:03:02.821 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: value unchanged 2024-01-11T05:03:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f528c030fa0 msgr2=0x7f528c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f528c030fa0 0x7f528c033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f52940096a0 tx=0x7f5294006b40).stop 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 msgr2=0x7f52a0108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a0108b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5290010d50 tx=0x7f5290006f70).stop 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 shutdown_connections 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f528c030fa0 0x7f528c033460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 --2- 172.21.15.67:0/1558619020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a00fc4d0 0x7f52a0108b60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 >> 172.21.15.67:0/1558619020 conn(0x7f52a00f5e20 msgr2=0x7f52a00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 shutdown_connections 2024-01-11T05:03:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:02.823+0000 7f52a682e700 1 -- 172.21.15.67:0/1558619020 wait complete. 2024-01-11T05:03:03.097 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:01.909025+0000 mgr.smithi067.srbzpc (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:03.097 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:02.005227+0000 mon.smithi067 (mon.0) 52 : cluster [DBG] mgrmap e8: smithi067.srbzpc(active, since 2s) 2024-01-11T05:03:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:Generating ssh key... 2024-01-11T05:03:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.528+0000 7f45cb928700 1 Processor -- start 2024-01-11T05:03:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.528+0000 7f45cb928700 1 -- start start 2024-01-11T05:03:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.528+0000 7f45cb928700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c4070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.528+0000 7f45cb928700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c4072080 con 0x7f45c4071c60 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45c96c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45c96c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c4070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48142/0 (socket says 172.21.15.67:48142) 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45c96c4700 1 -- 172.21.15.67:0/1373407744 learned_addr learned my addr 172.21.15.67:0/1373407744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45c96c4700 1 -- 172.21.15.67:0/1373407744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c4070650 con 0x7f45c4071c60 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45c96c4700 1 --2- 172.21.15.67:0/1373407744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c4070160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f45b400b5f0 tx=0x7f45b4006ee0).ready entity=mon.0 client_cookie=a565549ba998be4b server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45b3fff700 1 -- 172.21.15.67:0/1373407744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45b400d600 con 0x7f45c4071c60 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45b3fff700 1 -- 172.21.15.67:0/1373407744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f45b400dbc0 con 0x7f45c4071c60 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.529+0000 7f45b3fff700 1 -- 172.21.15.67:0/1373407744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45b4011b90 con 0x7f45c4071c60 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 -- 172.21.15.67:0/1373407744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 msgr2=0x7f45c4070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 --2- 172.21.15.67:0/1373407744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c4070160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f45b400b5f0 tx=0x7f45b4006ee0).stop 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 -- 172.21.15.67:0/1373407744 shutdown_connections 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 --2- 172.21.15.67:0/1373407744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c4070160 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 -- 172.21.15.67:0/1373407744 >> 172.21.15.67:0/1373407744 conn(0x7f45c40f7770 msgr2=0x7f45c40f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 -- 172.21.15.67:0/1373407744 shutdown_connections 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.530+0000 7f45cb928700 1 -- 172.21.15.67:0/1373407744 wait complete. 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45cb928700 1 Processor -- start 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45cb928700 1 -- start start 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45cb928700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45cb928700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c410d3c0 con 0x7f45c4071c60 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45c96c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45c96c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48144/0 (socket says 172.21.15.67:48144) 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45c96c4700 1 -- 172.21.15.67:0/327745528 learned_addr learned my addr 172.21.15.67:0/327745528 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45c96c4700 1 -- 172.21.15.67:0/327745528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b400b200 con 0x7f45c4071c60 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.531+0000 7f45c96c4700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c410ced0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f45b400e040 tx=0x7f45b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.532+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45b401b570 con 0x7f45c4071c60 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.532+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f45b401bb30 con 0x7f45c4071c60 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.532+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c410d5c0 con 0x7f45c4071c60 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.532+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45b401a820 con 0x7f45c4071c60 2024-01-11T05:03:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.532+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c410da50 con 0x7f45c4071c60 2024-01-11T05:03:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.533+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f45b402b070 con 0x7f45c4071c60 2024-01-11T05:03:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.533+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c4045bb0 con 0x7f45c4071c60 2024-01-11T05:03:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.533+0000 7f45b1ffb700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f45ac030fa0 0x7f45ac033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.533+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f45b4042df0 con 0x7f45c4071c60 2024-01-11T05:03:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.533+0000 7f45c8ec3700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f45ac030fa0 0x7f45ac033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.533+0000 7f45c8ec3700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f45ac030fa0 0x7f45ac033460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f45c00096a0 tx=0x7f45c0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:03.537 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.537+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f45b4046030 con 0x7f45c4071c60 2024-01-11T05:03:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:03.698+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f45c410e370 con 0x7f45ac030fa0 2024-01-11T05:03:04.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:04 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:03.700380+0000 mgr.smithi067.srbzpc (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2024-01-11T05:03:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.364+0000 7f45b1ffb700 1 -- 172.21.15.67:0/327745528 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45c410e370 con 0x7f45ac030fa0 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f45ac030fa0 msgr2=0x7f45ac033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f45ac030fa0 0x7f45ac033460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f45c00096a0 tx=0x7f45c0006b40).stop 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 msgr2=0x7f45c410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c410ced0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f45b400e040 tx=0x7f45b4006ee0).stop 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 shutdown_connections 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f45ac030fa0 0x7f45ac033460 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 --2- 172.21.15.67:0/327745528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45c4071c60 0x7f45c410ced0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 >> 172.21.15.67:0/327745528 conn(0x7f45c40f7770 msgr2=0x7f45c40f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 shutdown_connections 2024-01-11T05:03:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:04.366+0000 7f45cb928700 1 -- 172.21.15.67:0/327745528 wait complete. 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4759e700 1 Processor -- start 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4759e700 1 -- start start 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4759e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480ff000 con 0x7f8c4806ad00 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48150/0 (socket says 172.21.15.67:48150) 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.061+0000 7f8c4659c700 1 -- 172.21.15.67:0/3484119337 learned_addr learned my addr 172.21.15.67:0/3484119337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4659c700 1 -- 172.21.15.67:0/3484119337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c480ff140 con 0x7f8c4806ad00 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4659c700 1 --2- 172.21.15.67:0/3484119337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480feb10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8c3800b5f0 tx=0x7f8c38006ee0).ready entity=mon.0 client_cookie=67ea601e78f11727 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4559a700 1 -- 172.21.15.67:0/3484119337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c3800d600 con 0x7f8c4806ad00 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4559a700 1 -- 172.21.15.67:0/3484119337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8c3800dbc0 con 0x7f8c4806ad00 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4559a700 1 -- 172.21.15.67:0/3484119337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c38011b90 con 0x7f8c4806ad00 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4759e700 1 -- 172.21.15.67:0/3484119337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 msgr2=0x7f8c480feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.062+0000 7f8c4759e700 1 --2- 172.21.15.67:0/3484119337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480feb10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8c3800b5f0 tx=0x7f8c38006ee0).stop 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 -- 172.21.15.67:0/3484119337 shutdown_connections 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 --2- 172.21.15.67:0/3484119337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480feb10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 -- 172.21.15.67:0/3484119337 >> 172.21.15.67:0/3484119337 conn(0x7f8c480f64d0 msgr2=0x7f8c480f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 -- 172.21.15.67:0/3484119337 shutdown_connections 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 -- 172.21.15.67:0/3484119337 wait complete. 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 Processor -- start 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 -- start start 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4759e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480faa10 con 0x7f8c4806ad00 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.063+0000 7f8c4659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c4659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48160/0 (socket says 172.21.15.67:48160) 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c4659c700 1 -- 172.21.15.67:0/3529984943 learned_addr learned my addr 172.21.15.67:0/3529984943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c4659c700 1 -- 172.21.15.67:0/3529984943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3800b200 con 0x7f8c4806ad00 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c4659c700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480fc590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8c38006800 tx=0x7f8c38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c3800d9c0 con 0x7f8c4806ad00 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f8c3801b870 con 0x7f8c4806ad00 2024-01-11T05:03:05.070 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c480fabb0 con 0x7f8c4806ad00 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c3801a740 con 0x7f8c4806ad00 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.064+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c480fb040 con 0x7f8c4806ad00 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.065+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f8c3802b070 con 0x7f8c4806ad00 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.065+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c48045bb0 con 0x7f8c4806ad00 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.065+0000 7f8c36ffd700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8c2c030fa0 0x7f8c2c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.065+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f8c38042ce0 con 0x7f8c4806ad00 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.065+0000 7f8c45d9b700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8c2c030fa0 0x7f8c2c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.066+0000 7f8c45d9b700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8c2c030fa0 0x7f8c2c033460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8c3c0096a0 tx=0x7f8c3c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:05.072 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.070+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8c38022a70 con 0x7f8c4806ad00 2024-01-11T05:03:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.232+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c480fb280 con 0x7f8c2c030fa0 2024-01-11T05:03:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.233+0000 7f8c36ffd700 1 -- 172.21.15.67:0/3529984943 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f8c480fb280 con 0x7f8c2c030fa0 2024-01-11T05:03:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBpB38VCx/OpP3BJyxeQAwHkbLjge+TEnE+IOD2KjVixmHcis50d0HxHpewifyRY/gFw87FQO4mf/jdBk7mqXqopLqt8KDQFzplBwGq5BT7SjOgROzZUPHU0UYPtfOfZ0EOZguImUL6JnwwZMDmbCwbfchmwBfNzX/jgDrXoivcEGfpkfqpDvnYO5LxP+JyMC6MEcG/RuE0CRp0utUVoJ+5k/XCnGAJDeI2tAgqDlcp4vcOQl8QhFKrlhSfhM+p5oyegmpJa8vA9DxoNfrJhOHEaLR0DVOu9Zt6i8LCvDqt5JDJhnfLwpiqMtMeIM8fd5dwtogJOAdcXukg6lYgxpm/HsAg87NfZY9bu93AEGfx1+wLDeJWZRCj6k3yrcra9YkT0MeWDfIxoKeoUcFyrJeZ1uIMBAtHxkK38AT/0LZQot/KeeXROSZBVr3fDYI7IOiu5gCgs5n1CLVZvwUC/Sdmo9XZid7YfYhhTJPqv1rQ3uPLBf4WN7Uq6vFTlRpjxc= ceph-89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:03:05.235 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8c2c030fa0 msgr2=0x7f8c2c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:05.235 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8c2c030fa0 0x7f8c2c033460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8c3c0096a0 tx=0x7f8c3c006b40).stop 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 msgr2=0x7f8c480fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480fc590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8c38006800 tx=0x7f8c38006ee0).stop 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 shutdown_connections 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8c2c030fa0 0x7f8c2c033460 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 --2- 172.21.15.67:0/3529984943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c4806ad00 0x7f8c480fc590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 >> 172.21.15.67:0/3529984943 conn(0x7f8c480f64d0 msgr2=0x7f8c480f7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 shutdown_connections 2024-01-11T05:03:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.234+0000 7f8c4759e700 1 -- 172.21.15.67:0/3529984943 wait complete. 2024-01-11T05:03:05.505 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:04.362725+0000 mon.smithi067 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:05.505 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:04.364452+0000 mon.smithi067 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-11T05:03:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:Adding key to root@localhost authorized_keys... 2024-01-11T05:03:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:Adding host smithi067... 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2956a67700 1 Processor -- start 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2956a67700 1 -- start start 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2956a67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f29500febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2956a67700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29500ff0d0 con 0x7f295006ade0 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2954803700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f29500febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2954803700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f29500febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48168/0 (socket says 172.21.15.67:48168) 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.913+0000 7f2954803700 1 -- 172.21.15.67:0/4278487704 learned_addr learned my addr 172.21.15.67:0/4278487704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:05.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.914+0000 7f2954803700 1 -- 172.21.15.67:0/4278487704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29500ff210 con 0x7f295006ade0 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.914+0000 7f2954803700 1 --2- 172.21.15.67:0/4278487704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f29500febe0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f294400b5f0 tx=0x7f2944006ee0).ready entity=mon.0 client_cookie=173888f5a46dd295 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.914+0000 7f294f7fe700 1 -- 172.21.15.67:0/4278487704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294400d600 con 0x7f295006ade0 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.914+0000 7f294f7fe700 1 -- 172.21.15.67:0/4278487704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f294400dbc0 con 0x7f295006ade0 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.914+0000 7f294f7fe700 1 -- 172.21.15.67:0/4278487704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2944011b90 con 0x7f295006ade0 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 -- 172.21.15.67:0/4278487704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 msgr2=0x7f29500febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 --2- 172.21.15.67:0/4278487704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f29500febe0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f294400b5f0 tx=0x7f2944006ee0).stop 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 -- 172.21.15.67:0/4278487704 shutdown_connections 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 --2- 172.21.15.67:0/4278487704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f29500febe0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 -- 172.21.15.67:0/4278487704 >> 172.21.15.67:0/4278487704 conn(0x7f29500f6680 msgr2=0x7f29500f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 -- 172.21.15.67:0/4278487704 shutdown_connections 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 -- 172.21.15.67:0/4278487704 wait complete. 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 Processor -- start 2024-01-11T05:03:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.915+0000 7f2956a67700 1 -- start start 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2956a67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f295010ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2956a67700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295010b230 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2954803700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f295010ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2954803700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f295010ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48176/0 (socket says 172.21.15.67:48176) 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2954803700 1 -- 172.21.15.67:0/1716206013 learned_addr learned my addr 172.21.15.67:0/1716206013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2954803700 1 -- 172.21.15.67:0/1716206013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294400b200 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2954803700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f295010ad40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2944000c00 tx=0x7f2944006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294401b410 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295010b430 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f294401b9d0 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295010c040 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.916+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294401a620 con 0x7f295006ade0 2024-01-11T05:03:05.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.917+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f294402b070 con 0x7f295006ade0 2024-01-11T05:03:05.921 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.917+0000 7f294d7fa700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f2934030fa0 0x7f2934033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:05.921 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.917+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f2944042c10 con 0x7f295006ade0 2024-01-11T05:03:05.921 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.918+0000 7f294ffff700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f2934030fa0 0x7f2934033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:05.921 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.918+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29500591e0 con 0x7f295006ade0 2024-01-11T05:03:05.921 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.918+0000 7f294ffff700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f2934030fa0 0x7f2934033460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f293c0096a0 tx=0x7f293c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:05.922 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:05.922+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2944016070 con 0x7f295006ade0 2024-01-11T05:03:06.089 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.089+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}) v1 -- 0x7f29500fffe0 con 0x7f2934030fa0 2024-01-11T05:03:06.703 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Added host 'smithi067' with addr '172.21.15.67' 2024-01-11T05:03:06.703 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.702+0000 7f294d7fa700 1 -- 172.21.15.67:0/1716206013 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f29500fffe0 con 0x7f2934030fa0 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.704+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f2934030fa0 msgr2=0x7f2934033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.704+0000 7f2956a67700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f2934030fa0 0x7f2934033460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f293c0096a0 tx=0x7f293c006b40).stop 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.704+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 msgr2=0x7f295010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.704+0000 7f2956a67700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f295010ad40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2944000c00 tx=0x7f2944006ee0).stop 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.705+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 shutdown_connections 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.705+0000 7f2956a67700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f2934030fa0 0x7f2934033460 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.705+0000 7f2956a67700 1 --2- 172.21.15.67:0/1716206013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006ade0 0x7f295010ad40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.705+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 >> 172.21.15.67:0/1716206013 conn(0x7f29500f6680 msgr2=0x7f29500f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.705+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 shutdown_connections 2024-01-11T05:03:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:06.705+0000 7f2956a67700 1 -- 172.21.15.67:0/1716206013 wait complete. 2024-01-11T05:03:06.734 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:06 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:05.233654+0000 mgr.smithi067.srbzpc (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:07.027 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying mon service with default placement... 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.423+0000 7fcd84cdd700 1 Processor -- start 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.423+0000 7fcd84cdd700 1 -- start start 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd84cdd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd800fa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd84cdd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800fabf0 con 0x7fcd800fa2e0 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd7e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd800fa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd7e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd800fa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48186/0 (socket says 172.21.15.67:48186) 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd7e59c700 1 -- 172.21.15.67:0/2276223385 learned_addr learned my addr 172.21.15.67:0/2276223385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd7e59c700 1 -- 172.21.15.67:0/2276223385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd800fad30 con 0x7fcd800fa2e0 2024-01-11T05:03:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.424+0000 7fcd7e59c700 1 --2- 172.21.15.67:0/2276223385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd800fa700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcd7000b5f0 tx=0x7fcd70006ee0).ready entity=mon.0 client_cookie=213a7bf10854cc0c server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd7d59a700 1 -- 172.21.15.67:0/2276223385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd7000d600 con 0x7fcd800fa2e0 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd7d59a700 1 -- 172.21.15.67:0/2276223385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcd7000dbc0 con 0x7fcd800fa2e0 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd7d59a700 1 -- 172.21.15.67:0/2276223385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd70011b90 con 0x7fcd800fa2e0 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 -- 172.21.15.67:0/2276223385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 msgr2=0x7fcd800fa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 --2- 172.21.15.67:0/2276223385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd800fa700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcd7000b5f0 tx=0x7fcd70006ee0).stop 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 -- 172.21.15.67:0/2276223385 shutdown_connections 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 --2- 172.21.15.67:0/2276223385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd800fa700 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 -- 172.21.15.67:0/2276223385 >> 172.21.15.67:0/2276223385 conn(0x7fcd800f5e20 msgr2=0x7fcd800f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 -- 172.21.15.67:0/2276223385 shutdown_connections 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.425+0000 7fcd84cdd700 1 -- 172.21.15.67:0/2276223385 wait complete. 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd84cdd700 1 Processor -- start 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd84cdd700 1 -- start start 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd84cdd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd8010ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd84cdd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8010b220 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd7e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd8010ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd7e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd8010ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48188/0 (socket says 172.21.15.67:48188) 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.426+0000 7fcd7e59c700 1 -- 172.21.15.67:0/3505379767 learned_addr learned my addr 172.21.15.67:0/3505379767 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd7e59c700 1 -- 172.21.15.67:0/3505379767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7000b200 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd7e59c700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd8010ad30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcd7000e040 tx=0x7fcd70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd7000de00 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcd7001b870 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8010b420 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd7001a6c0 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.427+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8010c030 con 0x7fcd800fa2e0 2024-01-11T05:03:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.428+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fcd7000f070 con 0x7fcd800fa2e0 2024-01-11T05:03:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.428+0000 7fcd76ffd700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fcd60030ff0 0x7fcd600334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.428+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fcd70042ab0 con 0x7fcd800fa2e0 2024-01-11T05:03:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.428+0000 7fcd7dd9b700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fcd60030ff0 0x7fcd600334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.428+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd800591e0 con 0x7fcd800fa2e0 2024-01-11T05:03:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.429+0000 7fcd7dd9b700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fcd60030ff0 0x7fcd600334b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcd680096a0 tx=0x7fcd68006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.433+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcd7002b8c0 con 0x7fcd800fa2e0 2024-01-11T05:03:07.603 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.603+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd800fde50 con 0x7fcd60030ff0 2024-01-11T05:03:07.607 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.607+0000 7fcd76ffd700 1 -- 172.21.15.67:0/3505379767 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcd800fde50 con 0x7fcd60030ff0 2024-01-11T05:03:07.607 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fcd60030ff0 msgr2=0x7fcd600334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fcd60030ff0 0x7fcd600334b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcd680096a0 tx=0x7fcd68006b40).stop 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 msgr2=0x7fcd8010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd8010ad30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcd7000e040 tx=0x7fcd70006ee0).stop 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 shutdown_connections 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fcd60030ff0 0x7fcd600334b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 --2- 172.21.15.67:0/3505379767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd800fa2e0 0x7fcd8010ad30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 >> 172.21.15.67:0/3505379767 conn(0x7fcd800f5e20 msgr2=0x7fcd800ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:07.611 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 shutdown_connections 2024-01-11T05:03:07.611 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:07.609+0000 7fcd84cdd700 1 -- 172.21.15.67:0/3505379767 wait complete. 2024-01-11T05:03:07.953 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying mgr service with default placement... 2024-01-11T05:03:08.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:06.090367+0000 mgr.smithi067.srbzpc (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:08.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:06.702442+0000 mon.smithi067 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:08.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:06.702739+0000 mgr.smithi067.srbzpc (mgr.14116) 9 : cephadm [INF] Added host smithi067 2024-01-11T05:03:08.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:06.702991+0000 mon.smithi067 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:08.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:07.607663+0000 mon.smithi067 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.308+0000 7f64a0c0e700 1 Processor -- start 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.308+0000 7f64a0c0e700 1 -- start start 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.308+0000 7f64a0c0e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.308+0000 7f64a0c0e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0ff0d0 con 0x7f649c027670 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.308+0000 7f649a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48192/0 (socket says 172.21.15.67:48192) 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649a59c700 1 -- 172.21.15.67:0/3270647416 learned_addr learned my addr 172.21.15.67:0/3270647416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649a59c700 1 -- 172.21.15.67:0/3270647416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0ff210 con 0x7f649c027670 2024-01-11T05:03:08.310 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649a59c700 1 --2- 172.21.15.67:0/3270647416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0febe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f648400b5f0 tx=0x7f6484006ee0).ready entity=mon.0 client_cookie=3985ff34b254a2a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649959a700 1 -- 172.21.15.67:0/3270647416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648400d600 con 0x7f649c027670 2024-01-11T05:03:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649959a700 1 -- 172.21.15.67:0/3270647416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f648400dbc0 con 0x7f649c027670 2024-01-11T05:03:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.309+0000 7f649959a700 1 -- 172.21.15.67:0/3270647416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6484011b90 con 0x7f649c027670 2024-01-11T05:03:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 -- 172.21.15.67:0/3270647416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 msgr2=0x7f649c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 --2- 172.21.15.67:0/3270647416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0febe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f648400b5f0 tx=0x7f6484006ee0).stop 2024-01-11T05:03:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 -- 172.21.15.67:0/3270647416 shutdown_connections 2024-01-11T05:03:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 --2- 172.21.15.67:0/3270647416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0febe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 -- 172.21.15.67:0/3270647416 >> 172.21.15.67:0/3270647416 conn(0x7f649c0f6680 msgr2=0x7f649c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 -- 172.21.15.67:0/3270647416 shutdown_connections 2024-01-11T05:03:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 -- 172.21.15.67:0/3270647416 wait complete. 2024-01-11T05:03:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 Processor -- start 2024-01-11T05:03:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.310+0000 7f64a0c0e700 1 -- start start 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f64a0c0e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f64a0c0e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0faae0 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f649a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f649a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48206/0 (socket says 172.21.15.67:48206) 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f649a59c700 1 -- 172.21.15.67:0/2762129778 learned_addr learned my addr 172.21.15.67:0/2762129778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f649a59c700 1 -- 172.21.15.67:0/2762129778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648400b200 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f649a59c700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0fc660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6484006800 tx=0x7f6484006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648400de50 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c0fac80 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f648401b870 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f648401a7a0 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.311+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c0fb110 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.312+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f648400f070 con 0x7f649c027670 2024-01-11T05:03:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.312+0000 7f6492ffd700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f647c030fa0 0x7f647c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.312+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6484042c00 con 0x7f649c027670 2024-01-11T05:03:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.312+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c045bb0 con 0x7f649c027670 2024-01-11T05:03:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.312+0000 7f6499d9b700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f647c030fa0 0x7f647c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.313+0000 7f6499d9b700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f647c030fa0 0x7f647c033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f648c0096a0 tx=0x7f648c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.317+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6484016030 con 0x7f649c027670 2024-01-11T05:03:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.490+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f649c104c80 con 0x7f647c030fa0 2024-01-11T05:03:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.494+0000 7f6492ffd700 1 -- 172.21.15.67:0/2762129778 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f649c104c80 con 0x7f647c030fa0 2024-01-11T05:03:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f647c030fa0 msgr2=0x7f647c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f647c030fa0 0x7f647c033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f648c0096a0 tx=0x7f648c006b40).stop 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 msgr2=0x7f649c0fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0fc660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6484006800 tx=0x7f6484006ee0).stop 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 shutdown_connections 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f647c030fa0 0x7f647c033460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 --2- 172.21.15.67:0/2762129778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f649c027670 0x7f649c0fc660 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.496+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 >> 172.21.15.67:0/2762129778 conn(0x7f649c0f6680 msgr2=0x7f649c0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.497+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 shutdown_connections 2024-01-11T05:03:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:08.497+0000 7f64a0c0e700 1 -- 172.21.15.67:0/2762129778 wait complete. 2024-01-11T05:03:08.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:07.604637+0000 mgr.smithi067.srbzpc (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:08.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:07.605397+0000 mgr.smithi067.srbzpc (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-11T05:03:08.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:08.495034+0000 mon.smithi067 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:08.853 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying crash service with default placement... 2024-01-11T05:03:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc142313700 1 Processor -- start 2024-01-11T05:03:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc142313700 1 -- start start 2024-01-11T05:03:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc142313700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc142313700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0fca50 con 0x7fc13c0275f0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc13bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc13bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48208/0 (socket says 172.21.15.67:48208) 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.216+0000 7fc13bfff700 1 -- 172.21.15.67:0/2071498158 learned_addr learned my addr 172.21.15.67:0/2071498158 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.217+0000 7fc13bfff700 1 -- 172.21.15.67:0/2071498158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c101550 con 0x7fc13c0275f0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.217+0000 7fc13bfff700 1 --2- 172.21.15.67:0/2071498158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c101060 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc12c00b5f0 tx=0x7fc12c006ee0).ready entity=mon.0 client_cookie=4c0c829862731e30 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.217+0000 7fc13affd700 1 -- 172.21.15.67:0/2071498158 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc12c00d600 con 0x7fc13c0275f0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.217+0000 7fc13affd700 1 -- 172.21.15.67:0/2071498158 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc12c00dbc0 con 0x7fc13c0275f0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.217+0000 7fc13affd700 1 -- 172.21.15.67:0/2071498158 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc12c011b90 con 0x7fc13c0275f0 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 -- 172.21.15.67:0/2071498158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 msgr2=0x7fc13c101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 --2- 172.21.15.67:0/2071498158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c101060 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc12c00b5f0 tx=0x7fc12c006ee0).stop 2024-01-11T05:03:09.220 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 -- 172.21.15.67:0/2071498158 shutdown_connections 2024-01-11T05:03:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 --2- 172.21.15.67:0/2071498158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c101060 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 -- 172.21.15.67:0/2071498158 >> 172.21.15.67:0/2071498158 conn(0x7fc13c0f6680 msgr2=0x7fc13c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 -- 172.21.15.67:0/2071498158 shutdown_connections 2024-01-11T05:03:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.218+0000 7fc142313700 1 -- 172.21.15.67:0/2071498158 wait complete. 2024-01-11T05:03:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc142313700 1 Processor -- start 2024-01-11T05:03:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc142313700 1 -- start start 2024-01-11T05:03:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc142313700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc142313700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10d3f0 con 0x7fc13c0275f0 2024-01-11T05:03:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc13bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc13bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48212/0 (socket says 172.21.15.67:48212) 2024-01-11T05:03:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc13bfff700 1 -- 172.21.15.67:0/3271762940 learned_addr learned my addr 172.21.15.67:0/3271762940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.219+0000 7fc13bfff700 1 -- 172.21.15.67:0/3271762940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c00b200 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.220+0000 7fc13bfff700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c10cf00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc12c000c00 tx=0x7fc12c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.220+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc12c00de00 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.220+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc12c01b870 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.220+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc12c01a670 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.220+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c10d5f0 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.220+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c10da80 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.221+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c045bb0 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.221+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fc12c02b070 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.221+0000 7fc138ff9700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fc128030fa0 0x7fc128033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.221+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fc12c042cd0 con 0x7fc13c0275f0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.221+0000 7fc13b7fe700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fc128030fa0 0x7fc128033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.222+0000 7fc13b7fe700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fc128030fa0 0x7fc128033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc1300096a0 tx=0x7fc130006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:09.225 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.225+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc12c01b9e0 con 0x7fc13c0275f0 2024-01-11T05:03:09.395 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.394+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fc13c10e2e0 con 0x7fc128030fa0 2024-01-11T05:03:09.399 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.399+0000 7fc138ff9700 1 -- 172.21.15.67:0/3271762940 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fc13c10e2e0 con 0x7fc128030fa0 2024-01-11T05:03:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-11T05:03:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fc128030fa0 msgr2=0x7fc128033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fc128030fa0 0x7fc128033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc1300096a0 tx=0x7fc130006b40).stop 2024-01-11T05:03:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 msgr2=0x7fc13c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c10cf00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc12c000c00 tx=0x7fc12c006ee0).stop 2024-01-11T05:03:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 shutdown_connections 2024-01-11T05:03:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fc128030fa0 0x7fc128033460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 --2- 172.21.15.67:0/3271762940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13c0275f0 0x7fc13c10cf00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 >> 172.21.15.67:0/3271762940 conn(0x7fc13c0f6680 msgr2=0x7fc13c0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 shutdown_connections 2024-01-11T05:03:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:09.401+0000 7fc142313700 1 -- 172.21.15.67:0/3271762940 wait complete. 2024-01-11T05:03:09.744 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying prometheus service with default placement... 2024-01-11T05:03:10.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:08.492161+0000 mgr.smithi067.srbzpc (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:10.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:08.492714+0000 mgr.smithi067.srbzpc (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-11T05:03:10.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:09.400114+0000 mon.smithi067 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:10.101 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a539b8700 1 Processor -- start 2024-01-11T05:03:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a539b8700 1 -- start start 2024-01-11T05:03:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a539b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a539b8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0ff0d0 con 0x7f0a4c027670 2024-01-11T05:03:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a51754700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a51754700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48228/0 (socket says 172.21.15.67:48228) 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a51754700 1 -- 172.21.15.67:0/4082581827 learned_addr learned my addr 172.21.15.67:0/4082581827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.095+0000 7f0a51754700 1 -- 172.21.15.67:0/4082581827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0ff210 con 0x7f0a4c027670 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a51754700 1 --2- 172.21.15.67:0/4082581827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c0febe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0a4800b5f0 tx=0x7f0a48006ee0).ready entity=mon.0 client_cookie=7101ef3c27fc9781 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a3bfff700 1 -- 172.21.15.67:0/4082581827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a4800d600 con 0x7f0a4c027670 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a3bfff700 1 -- 172.21.15.67:0/4082581827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0a4800dbc0 con 0x7f0a4c027670 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a3bfff700 1 -- 172.21.15.67:0/4082581827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a48011b90 con 0x7f0a4c027670 2024-01-11T05:03:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 -- 172.21.15.67:0/4082581827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 msgr2=0x7f0a4c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 --2- 172.21.15.67:0/4082581827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c0febe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0a4800b5f0 tx=0x7f0a48006ee0).stop 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 -- 172.21.15.67:0/4082581827 shutdown_connections 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 --2- 172.21.15.67:0/4082581827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c0febe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 -- 172.21.15.67:0/4082581827 >> 172.21.15.67:0/4082581827 conn(0x7f0a4c0f6680 msgr2=0x7f0a4c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 -- 172.21.15.67:0/4082581827 shutdown_connections 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.096+0000 7f0a539b8700 1 -- 172.21.15.67:0/4082581827 wait complete. 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a539b8700 1 Processor -- start 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a539b8700 1 -- start start 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a539b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a539b8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c10d3b0 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a51754700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a51754700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48242/0 (socket says 172.21.15.67:48242) 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.097+0000 7f0a51754700 1 -- 172.21.15.67:0/2720242802 learned_addr learned my addr 172.21.15.67:0/2720242802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a51754700 1 -- 172.21.15.67:0/2720242802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4800b200 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a51754700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c10cec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0a48006800 tx=0x7f0a48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a4800de50 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0a4801b870 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4c10d5b0 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a4801a7a0 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.098+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4c10da40 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.099+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f0a4800f070 con 0x7f0a4c027670 2024-01-11T05:03:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.099+0000 7f0a39ffb700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f0a34030ff0 0x7f0a340334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.099+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0a48042bb0 con 0x7f0a4c027670 2024-01-11T05:03:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.099+0000 7f0a50f53700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f0a34030ff0 0x7f0a340334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.100+0000 7f0a50f53700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f0a34030ff0 0x7f0a340334b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0a3c0096a0 tx=0x7f0a3c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.100+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4c045bb0 con 0x7f0a4c027670 2024-01-11T05:03:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.104+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0a48016030 con 0x7f0a4c027670 2024-01-11T05:03:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.274+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a4c0008d0 con 0x7f0a34030ff0 2024-01-11T05:03:10.278 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.277+0000 7f0a39ffb700 1 -- 172.21.15.67:0/2720242802 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f0a4c0008d0 con 0x7f0a34030ff0 2024-01-11T05:03:10.278 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-11T05:03:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f0a34030ff0 msgr2=0x7f0a340334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f0a34030ff0 0x7f0a340334b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0a3c0096a0 tx=0x7f0a3c006b40).stop 2024-01-11T05:03:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 msgr2=0x7f0a4c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c10cec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0a48006800 tx=0x7f0a48006ee0).stop 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 shutdown_connections 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f0a34030ff0 0x7f0a340334b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 --2- 172.21.15.67:0/2720242802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4c027670 0x7f0a4c10cec0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 >> 172.21.15.67:0/2720242802 conn(0x7f0a4c0f6680 msgr2=0x7f0a4c0f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 shutdown_connections 2024-01-11T05:03:10.281 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.279+0000 7f0a539b8700 1 -- 172.21.15.67:0/2720242802 wait complete. 2024-01-11T05:03:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying grafana service with default placement... 2024-01-11T05:03:10.983 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.978+0000 7f6b92fcb700 1 Processor -- start 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.978+0000 7f6b92fcb700 1 -- start start 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.978+0000 7f6b92fcb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c070100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.978+0000 7f6b92fcb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c072020 con 0x7f6b8c071c00 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b90d67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b90d67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48244/0 (socket says 172.21.15.67:48244) 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b90d67700 1 -- 172.21.15.67:0/2142311222 learned_addr learned my addr 172.21.15.67:0/2142311222 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b90d67700 1 -- 172.21.15.67:0/2142311222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c0705f0 con 0x7f6b8c071c00 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b90d67700 1 --2- 172.21.15.67:0/2142311222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c070100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6b8000b5f0 tx=0x7f6b80006ee0).ready entity=mon.0 client_cookie=3be9bf4cee767403 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b8b7fe700 1 -- 172.21.15.67:0/2142311222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b8000d600 con 0x7f6b8c071c00 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b8b7fe700 1 -- 172.21.15.67:0/2142311222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f6b8000dbc0 con 0x7f6b8c071c00 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.979+0000 7f6b8b7fe700 1 -- 172.21.15.67:0/2142311222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b80011b90 con 0x7f6b8c071c00 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 -- 172.21.15.67:0/2142311222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 msgr2=0x7f6b8c070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 --2- 172.21.15.67:0/2142311222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c070100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6b8000b5f0 tx=0x7f6b80006ee0).stop 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 -- 172.21.15.67:0/2142311222 shutdown_connections 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 --2- 172.21.15.67:0/2142311222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c070100 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 -- 172.21.15.67:0/2142311222 >> 172.21.15.67:0/2142311222 conn(0x7f6b8c0f7770 msgr2=0x7f6b8c0f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 -- 172.21.15.67:0/2142311222 shutdown_connections 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 -- 172.21.15.67:0/2142311222 wait complete. 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.980+0000 7f6b92fcb700 1 Processor -- start 2024-01-11T05:03:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b92fcb700 1 -- start start 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b92fcb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c10cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b92fcb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c10d460 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b90d67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c10cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b90d67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c10cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48248/0 (socket says 172.21.15.67:48248) 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b90d67700 1 -- 172.21.15.67:0/1487471084 learned_addr learned my addr 172.21.15.67:0/1487471084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b90d67700 1 -- 172.21.15.67:0/1487471084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8000b200 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.981+0000 7f6b90d67700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c10cf70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6b80000c00 tx=0x7f6b80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.982+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b80011560 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.982+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f6b8001b870 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.982+0000 7f6b92fcb700 1 -- 172.21.15.67:0/1487471084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c10d660 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.982+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b8001a690 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.982+0000 7f6b92fcb700 1 -- 172.21.15.67:0/1487471084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c10daf0 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.982+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f6b8002b070 con 0x7f6b8c071c00 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.983+0000 7f6b897fa700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f6b70030fa0 0x7f6b70033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.983+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6b80042dc0 con 0x7f6b8c071c00 2024-01-11T05:03:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.983+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b64004fa0 con 0x7f6b8c071c00 2024-01-11T05:03:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.983+0000 7f6b8bfff700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f6b70030fa0 0x7f6b70033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.983+0000 7f6b8bfff700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f6b70030fa0 0x7f6b70033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6b780096a0 tx=0x7f6b78006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:10.987+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6b80016080 con 0x7f6b8c071c00 2024-01-11T05:03:11.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:10 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:09.396395+0000 mgr.smithi067.srbzpc (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:11.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:10 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:09.397065+0000 mgr.smithi067.srbzpc (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-11T05:03:11.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:10 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:10.278246+0000 mon.smithi067 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.157+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b64000bc0 con 0x7f6b70030fa0 2024-01-11T05:03:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.160+0000 7f6b897fa700 1 -- 172.21.15.67:0/1487471084 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f6b64000bc0 con 0x7f6b70030fa0 2024-01-11T05:03:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f6b70030fa0 msgr2=0x7f6b70033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f6b70030fa0 0x7f6b70033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6b780096a0 tx=0x7f6b78006b40).stop 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 msgr2=0x7f6b8c10cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c10cf70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6b80000c00 tx=0x7f6b80006ee0).stop 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 shutdown_connections 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f6b70030fa0 0x7f6b70033460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 --2- 172.21.15.67:0/1487471084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c071c00 0x7f6b8c10cf70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 >> 172.21.15.67:0/1487471084 conn(0x7f6b8c0f7770 msgr2=0x7f6b8c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 shutdown_connections 2024-01-11T05:03:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.162+0000 7f6b76ffd700 1 -- 172.21.15.67:0/1487471084 wait complete. 2024-01-11T05:03:11.541 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying node-exporter service with default placement... 2024-01-11T05:03:11.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:10.275559+0000 mgr.smithi067.srbzpc (mgr.14116) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:10.276113+0000 mgr.smithi067.srbzpc (mgr.14116) 17 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-11T05:03:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:11.161259+0000 mon.smithi067 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.882+0000 7faec7cf2700 1 Processor -- start 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec7cf2700 1 -- start start 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec7cf2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec00fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec7cf2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec00fb7e0 con 0x7faec00faed0 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec5a8e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec00fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec5a8e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec00fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48252/0 (socket says 172.21.15.67:48252) 2024-01-11T05:03:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec5a8e700 1 -- 172.21.15.67:0/3761628817 learned_addr learned my addr 172.21.15.67:0/3761628817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec5a8e700 1 -- 172.21.15.67:0/3761628817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00fb920 con 0x7faec00faed0 2024-01-11T05:03:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.883+0000 7faec5a8e700 1 --2- 172.21.15.67:0/3761628817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec00fb2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7faebc00b5f0 tx=0x7faebc006ee0).ready entity=mon.0 client_cookie=73667b9d4ce312a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec4a8c700 1 -- 172.21.15.67:0/3761628817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faebc00d600 con 0x7faec00faed0 2024-01-11T05:03:11.887 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec4a8c700 1 -- 172.21.15.67:0/3761628817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7faebc00dbc0 con 0x7faec00faed0 2024-01-11T05:03:11.887 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec4a8c700 1 -- 172.21.15.67:0/3761628817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faebc011b90 con 0x7faec00faed0 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 -- 172.21.15.67:0/3761628817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 msgr2=0x7faec00fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 --2- 172.21.15.67:0/3761628817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec00fb2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7faebc00b5f0 tx=0x7faebc006ee0).stop 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 -- 172.21.15.67:0/3761628817 shutdown_connections 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 --2- 172.21.15.67:0/3761628817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec00fb2f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 -- 172.21.15.67:0/3761628817 >> 172.21.15.67:0/3761628817 conn(0x7faec00f6680 msgr2=0x7faec00f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 -- 172.21.15.67:0/3761628817 shutdown_connections 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.884+0000 7faec7cf2700 1 -- 172.21.15.67:0/3761628817 wait complete. 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec7cf2700 1 Processor -- start 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec7cf2700 1 -- start start 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec7cf2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec7cf2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec010d3d0 con 0x7faec00faed0 2024-01-11T05:03:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec5a8e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec5a8e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec010cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48268/0 (socket says 172.21.15.67:48268) 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.885+0000 7faec5a8e700 1 -- 172.21.15.67:0/4160882228 learned_addr learned my addr 172.21.15.67:0/4160882228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faec5a8e700 1 -- 172.21.15.67:0/4160882228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc00b200 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faec5a8e700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec010cee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7faebc006800 tx=0x7faebc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faebc00de50 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec010d5d0 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec010da60 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7faebc01b870 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.886+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faebc01a7a0 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.887+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7faebc00f070 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.887+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea0004fa0 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.887+0000 7faeb27fc700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7faea8030ff0 0x7faea80334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.887+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7faebc042b50 con 0x7faec00faed0 2024-01-11T05:03:11.889 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.887+0000 7faec528d700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7faea8030ff0 0x7faea80334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:11.890 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.887+0000 7faec528d700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7faea8030ff0 0x7faea80334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7faeb40096a0 tx=0x7faeb4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:11.891 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:11.891+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faebc02b4d0 con 0x7faec00faed0 2024-01-11T05:03:12.061 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.061+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7faea0000bc0 con 0x7faea8030ff0 2024-01-11T05:03:12.065 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.064+0000 7faeb27fc700 1 -- 172.21.15.67:0/4160882228 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7faea0000bc0 con 0x7faea8030ff0 2024-01-11T05:03:12.065 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-11T05:03:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7faea8030ff0 msgr2=0x7faea80334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7faea8030ff0 0x7faea80334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7faeb40096a0 tx=0x7faeb4006b40).stop 2024-01-11T05:03:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 msgr2=0x7faec010cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec010cee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7faebc006800 tx=0x7faebc006ee0).stop 2024-01-11T05:03:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 shutdown_connections 2024-01-11T05:03:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7faea8030ff0 0x7faea80334b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 --2- 172.21.15.67:0/4160882228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faec00faed0 0x7faec010cee0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 >> 172.21.15.67:0/4160882228 conn(0x7faec00f6680 msgr2=0x7faec00271f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 shutdown_connections 2024-01-11T05:03:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.066+0000 7faec7cf2700 1 -- 172.21.15.67:0/4160882228 wait complete. 2024-01-11T05:03:12.419 INFO:teuthology.orchestra.run.smithi067.stderr:Deploying alertmanager service with default placement... 2024-01-11T05:03:12.774 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4da35700 1 Processor -- start 2024-01-11T05:03:12.774 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4da35700 1 -- start start 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4da35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef480fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4da35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480fef20 con 0x7fef4806ade0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4ca33700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef480fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4ca33700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef480fea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41760/0 (socket says 172.21.15.67:41760) 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4ca33700 1 -- 172.21.15.67:0/3062383061 learned_addr learned my addr 172.21.15.67:0/3062383061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.769+0000 7fef4ca33700 1 -- 172.21.15.67:0/3062383061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480ff060 con 0x7fef4806ade0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4ca33700 1 --2- 172.21.15.67:0/3062383061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef480fea30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fef3c00b5f0 tx=0x7fef3c006ee0).ready entity=mon.0 client_cookie=917564de9e8f6490 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef477fe700 1 -- 172.21.15.67:0/3062383061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c00d600 con 0x7fef4806ade0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef477fe700 1 -- 172.21.15.67:0/3062383061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fef3c00dbc0 con 0x7fef4806ade0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef477fe700 1 -- 172.21.15.67:0/3062383061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c011b90 con 0x7fef4806ade0 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 -- 172.21.15.67:0/3062383061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 msgr2=0x7fef480fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 --2- 172.21.15.67:0/3062383061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef480fea30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fef3c00b5f0 tx=0x7fef3c006ee0).stop 2024-01-11T05:03:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 -- 172.21.15.67:0/3062383061 shutdown_connections 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 --2- 172.21.15.67:0/3062383061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef480fea30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 -- 172.21.15.67:0/3062383061 >> 172.21.15.67:0/3062383061 conn(0x7fef480f64d0 msgr2=0x7fef480f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 -- 172.21.15.67:0/3062383061 shutdown_connections 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.770+0000 7fef4da35700 1 -- 172.21.15.67:0/3062383061 wait complete. 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4da35700 1 Processor -- start 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4da35700 1 -- start start 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4da35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef4810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4da35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4810b010 con 0x7fef4806ade0 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4ca33700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef4810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4ca33700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef4810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41772/0 (socket says 172.21.15.67:41772) 2024-01-11T05:03:12.776 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.771+0000 7fef4ca33700 1 -- 172.21.15.67:0/4250128109 learned_addr learned my addr 172.21.15.67:0/4250128109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef4ca33700 1 -- 172.21.15.67:0/4250128109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c00b200 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef4ca33700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef4810ab20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fef3c006800 tx=0x7fef3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c00de50 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fef3c01b870 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4810b210 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c01a7a0 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.772+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4810b6a0 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.773+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fef3c00f070 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.773+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef48045bb0 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.773+0000 7fef457fa700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fef2c030fa0 0x7fef2c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.773+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fef3c042c20 con 0x7fef4806ade0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.773+0000 7fef47fff700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fef2c030fa0 0x7fef2c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.774+0000 7fef47fff700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fef2c030fa0 0x7fef2c033460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fef340096a0 tx=0x7fef34006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:12.778 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.777+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fef3c019cd0 con 0x7fef4806ade0 2024-01-11T05:03:12.945 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.945+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fef4810bf00 con 0x7fef2c030fa0 2024-01-11T05:03:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.948+0000 7fef457fa700 1 -- 172.21.15.67:0/4250128109 <== mgr.14116 v2:172.21.15.67:6800/3176394990 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fef4810bf00 con 0x7fef2c030fa0 2024-01-11T05:03:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-11T05:03:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fef2c030fa0 msgr2=0x7fef2c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fef2c030fa0 0x7fef2c033460 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fef340096a0 tx=0x7fef34006b40).stop 2024-01-11T05:03:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 msgr2=0x7fef4810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef4810ab20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fef3c006800 tx=0x7fef3c006ee0).stop 2024-01-11T05:03:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 shutdown_connections 2024-01-11T05:03:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fef2c030fa0 0x7fef2c033460 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 --2- 172.21.15.67:0/4250128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef4806ade0 0x7fef4810ab20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 >> 172.21.15.67:0/4250128109 conn(0x7fef480f64d0 msgr2=0x7fef480f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 shutdown_connections 2024-01-11T05:03:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:12.950+0000 7fef4da35700 1 -- 172.21.15.67:0/4250128109 wait complete. 2024-01-11T05:03:12.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:12 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:11.158487+0000 mgr.smithi067.srbzpc (mgr.14116) 18 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:12.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:12 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:11.158993+0000 mgr.smithi067.srbzpc (mgr.14116) 19 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-11T05:03:12.979 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:12 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:12.065282+0000 mon.smithi067 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.694+0000 7fbfea1a0700 1 Processor -- start 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.694+0000 7fbfea1a0700 1 -- start start 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.694+0000 7fbfea1a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe4070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.694+0000 7fbfea1a0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4072060 con 0x7fbfe4071c40 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.694+0000 7fbfe919e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe4070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfe919e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe4070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41774/0 (socket says 172.21.15.67:41774) 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfe919e700 1 -- 172.21.15.67:0/3300086403 learned_addr learned my addr 172.21.15.67:0/3300086403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfe919e700 1 -- 172.21.15.67:0/3300086403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4070630 con 0x7fbfe4071c40 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfe919e700 1 --2- 172.21.15.67:0/3300086403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe4070140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fbfd400b5f0 tx=0x7fbfd4006ee0).ready entity=mon.0 client_cookie=fc2d57d899134489 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:13.697 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfd3fff700 1 -- 172.21.15.67:0/3300086403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd400d600 con 0x7fbfe4071c40 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfd3fff700 1 -- 172.21.15.67:0/3300086403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fbfd400dbc0 con 0x7fbfe4071c40 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.695+0000 7fbfd3fff700 1 -- 172.21.15.67:0/3300086403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd4011b90 con 0x7fbfe4071c40 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 -- 172.21.15.67:0/3300086403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 msgr2=0x7fbfe4070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 --2- 172.21.15.67:0/3300086403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe4070140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fbfd400b5f0 tx=0x7fbfd4006ee0).stop 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 -- 172.21.15.67:0/3300086403 shutdown_connections 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 --2- 172.21.15.67:0/3300086403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe4070140 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 -- 172.21.15.67:0/3300086403 >> 172.21.15.67:0/3300086403 conn(0x7fbfe40f7600 msgr2=0x7fbfe40f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 -- 172.21.15.67:0/3300086403 shutdown_connections 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 -- 172.21.15.67:0/3300086403 wait complete. 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.696+0000 7fbfea1a0700 1 Processor -- start 2024-01-11T05:03:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfea1a0700 1 -- start start 2024-01-11T05:03:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfea1a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe410ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfea1a0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe410d310 con 0x7fbfe4071c40 2024-01-11T05:03:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfe919e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe410ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfe919e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe410ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41782/0 (socket says 172.21.15.67:41782) 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfe919e700 1 -- 172.21.15.67:0/2395733901 learned_addr learned my addr 172.21.15.67:0/2395733901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfe919e700 1 -- 172.21.15.67:0/2395733901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd400b200 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfe919e700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe410ce20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbfd400c010 tx=0x7fbfd4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd400de90 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fbfd401b870 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe410d510 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.697+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd401a670 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.698+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe410e120 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.698+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fbfd400f070 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.699+0000 7fbfd1ffb700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fbfcc030ff0 0x7fbfcc0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.699+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fbfd4042ab0 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.699+0000 7fbfe899d700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fbfcc030ff0 0x7fbfcc0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.699+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe40591e0 con 0x7fbfe4071c40 2024-01-11T05:03:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.699+0000 7fbfe899d700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fbfcc030ff0 0x7fbfcc0334b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbfe00096a0 tx=0x7fbfe0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.703+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbfd4016030 con 0x7fbfe4071c40 2024-01-11T05:03:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.868+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fbfe4045bb0 con 0x7fbfe4071c40 2024-01-11T05:03:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.871+0000 7fbfd1ffb700 1 -- 172.21.15.67:0/2395733901 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fbfd40320d0 con 0x7fbfe4071c40 2024-01-11T05:03:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.873+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fbfcc030ff0 msgr2=0x7fbfcc0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.873+0000 7fbfea1a0700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fbfcc030ff0 0x7fbfcc0334b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbfe00096a0 tx=0x7fbfe0006b40).stop 2024-01-11T05:03:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.873+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 msgr2=0x7fbfe410ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.873+0000 7fbfea1a0700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe410ce20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbfd400c010 tx=0x7fbfd4006ee0).stop 2024-01-11T05:03:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.875+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 shutdown_connections 2024-01-11T05:03:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.875+0000 7fbfea1a0700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7fbfcc030ff0 0x7fbfcc0334b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.875+0000 7fbfea1a0700 1 --2- 172.21.15.67:0/2395733901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe4071c40 0x7fbfe410ce20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.875+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 >> 172.21.15.67:0/2395733901 conn(0x7fbfe40f7600 msgr2=0x7fbfe40f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.875+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 shutdown_connections 2024-01-11T05:03:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:13.875+0000 7fbfea1a0700 1 -- 172.21.15.67:0/2395733901 wait complete. 2024-01-11T05:03:14.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:12.062402+0000 mgr.smithi067.srbzpc (mgr.14116) 20 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:14.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:12.062923+0000 mgr.smithi067.srbzpc (mgr.14116) 21 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-11T05:03:14.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:12.949255+0000 mon.smithi067 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc8c083700 1 Processor -- start 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc8c083700 1 -- start start 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc8c083700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc84076310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc8c083700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84076800 con 0x7efc84075ef0 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc89e1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc84076310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc89e1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc84076310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41792/0 (socket says 172.21.15.67:41792) 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.845+0000 7efc89e1f700 1 -- 172.21.15.67:0/2716817356 learned_addr learned my addr 172.21.15.67:0/2716817356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.846+0000 7efc89e1f700 1 -- 172.21.15.67:0/2716817356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84076940 con 0x7efc84075ef0 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.846+0000 7efc89e1f700 1 --2- 172.21.15.67:0/2716817356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc84076310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7efc8000b5f0 tx=0x7efc80006ee0).ready entity=mon.0 client_cookie=73124e7fa177ada2 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.846+0000 7efc88e1d700 1 -- 172.21.15.67:0/2716817356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc8000d600 con 0x7efc84075ef0 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.846+0000 7efc88e1d700 1 -- 172.21.15.67:0/2716817356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efc8000dbf0 con 0x7efc84075ef0 2024-01-11T05:03:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.846+0000 7efc88e1d700 1 -- 172.21.15.67:0/2716817356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc80011bf0 con 0x7efc84075ef0 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 -- 172.21.15.67:0/2716817356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 msgr2=0x7efc84076310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 --2- 172.21.15.67:0/2716817356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc84076310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7efc8000b5f0 tx=0x7efc80006ee0).stop 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 -- 172.21.15.67:0/2716817356 shutdown_connections 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 --2- 172.21.15.67:0/2716817356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc84076310 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 -- 172.21.15.67:0/2716817356 >> 172.21.15.67:0/2716817356 conn(0x7efc840717f0 msgr2=0x7efc84073c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 -- 172.21.15.67:0/2716817356 shutdown_connections 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.847+0000 7efc8c083700 1 -- 172.21.15.67:0/2716817356 wait complete. 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc8c083700 1 Processor -- start 2024-01-11T05:03:14.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc8c083700 1 -- start start 2024-01-11T05:03:14.850 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc8c083700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc841113a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:14.850 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc8c083700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84111890 con 0x7efc84075ef0 2024-01-11T05:03:14.850 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc89e1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc841113a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:14.850 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc89e1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc841113a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41794/0 (socket says 172.21.15.67:41794) 2024-01-11T05:03:14.850 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc89e1f700 1 -- 172.21.15.67:0/2248337594 learned_addr learned my addr 172.21.15.67:0/2248337594 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc89e1f700 1 -- 172.21.15.67:0/2248337594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8000b200 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.848+0000 7efc89e1f700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc841113a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efc80006800 tx=0x7efc80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.849+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc8000dec0 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.849+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efc8001b8a0 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.849+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc8001a700 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.849+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc84111a90 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.849+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc84111f20 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.850+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7efc8002b070 con 0x7efc84075ef0 2024-01-11T05:03:14.851 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.850+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc84045bb0 con 0x7efc84075ef0 2024-01-11T05:03:14.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.850+0000 7efc727fc700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7efc6c030ff0 0x7efc6c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:14.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.850+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7efc80042b90 con 0x7efc84075ef0 2024-01-11T05:03:14.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.851+0000 7efc8961e700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7efc6c030ff0 0x7efc6c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:14.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.852+0000 7efc8961e700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7efc6c030ff0 0x7efc6c0334b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7efc740096a0 tx=0x7efc74006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:14.855 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:14.855+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efc80022b00 con 0x7efc84075ef0 2024-01-11T05:03:15.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:14 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:12.946582+0000 mgr.smithi067.srbzpc (mgr.14116) 22 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:15.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:14 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:12.947097+0000 mgr.smithi067.srbzpc (mgr.14116) 23 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-11T05:03:15.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:14 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:13.872537+0000 mon.smithi067 (mon.0) 64 : audit [INF] from='client.? 172.21.15.67:0/2395733901' entity='client.admin' 2024-01-11T05:03:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.015+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7efc84106560 con 0x7efc84075ef0 2024-01-11T05:03:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.020+0000 7efc727fc700 1 -- 172.21.15.67:0/2248337594 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7efc80016030 con 0x7efc84075ef0 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7efc6c030ff0 msgr2=0x7efc6c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7efc6c030ff0 0x7efc6c0334b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7efc740096a0 tx=0x7efc74006b40).stop 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 msgr2=0x7efc841113a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc841113a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efc80006800 tx=0x7efc80006ee0).stop 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 shutdown_connections 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7efc6c030ff0 0x7efc6c0334b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 --2- 172.21.15.67:0/2248337594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc84075ef0 0x7efc841113a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.022+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 >> 172.21.15.67:0/2248337594 conn(0x7efc840717f0 msgr2=0x7efc84073350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.023+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 shutdown_connections 2024-01-11T05:03:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.023+0000 7efc8c083700 1 -- 172.21.15.67:0/2248337594 wait complete. 2024-01-11T05:03:15.460 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling the dashboard module... 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f91e4f700 1 Processor -- start 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f91e4f700 1 -- start start 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f91e4f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f91e4f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c0730d0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f8b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f8b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c071250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41796/0 (socket says 172.21.15.67:41796) 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.816+0000 7f7f8b7fe700 1 -- 172.21.15.67:0/2754487137 learned_addr learned my addr 172.21.15.67:0/2754487137 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.817+0000 7f7f8b7fe700 1 -- 172.21.15.67:0/2754487137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c071740 con 0x7f7f8c072cb0 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.817+0000 7f7f8b7fe700 1 --2- 172.21.15.67:0/2754487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c071250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7f7c00b5f0 tx=0x7f7f7c006ee0).ready entity=mon.0 client_cookie=db71e46db6d3c362 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.817+0000 7f7f8a7fc700 1 -- 172.21.15.67:0/2754487137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f7c00d600 con 0x7f7f8c072cb0 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.817+0000 7f7f8a7fc700 1 -- 172.21.15.67:0/2754487137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7f7c00dbf0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.817+0000 7f7f8a7fc700 1 -- 172.21.15.67:0/2754487137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f7c011bf0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 -- 172.21.15.67:0/2754487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 msgr2=0x7f7f8c071250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 --2- 172.21.15.67:0/2754487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c071250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7f7c00b5f0 tx=0x7f7f7c006ee0).stop 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 -- 172.21.15.67:0/2754487137 shutdown_connections 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 --2- 172.21.15.67:0/2754487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c071250 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 -- 172.21.15.67:0/2754487137 >> 172.21.15.67:0/2754487137 conn(0x7f7f8c0f7780 msgr2=0x7f7f8c0f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 -- 172.21.15.67:0/2754487137 shutdown_connections 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 -- 172.21.15.67:0/2754487137 wait complete. 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.818+0000 7f7f91e4f700 1 Processor -- start 2024-01-11T05:03:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f91e4f700 1 -- start start 2024-01-11T05:03:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f91e4f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c10cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f91e4f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c10d440 con 0x7f7f8c072cb0 2024-01-11T05:03:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f8b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c10cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f8b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c10cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41810/0 (socket says 172.21.15.67:41810) 2024-01-11T05:03:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f8b7fe700 1 -- 172.21.15.67:0/790582510 learned_addr learned my addr 172.21.15.67:0/790582510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.819+0000 7f7f8b7fe700 1 -- 172.21.15.67:0/790582510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c00b200 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.820+0000 7f7f8b7fe700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c10cf50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7f7c006800 tx=0x7f7f7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.820+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f7c00dec0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.820+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7f7c01b8a0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.820+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c10d640 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.820+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f7c01a690 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.820+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c10dad0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.821+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c045bb0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.821+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f7f7c02b070 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.821+0000 7f7f73fff700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f7f78030fa0 0x7f7f78033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.821+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f7f7c042bf0 con 0x7f7f8c072cb0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.821+0000 7f7f8affd700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f7f78030fa0 0x7f7f78033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.822+0000 7f7f8affd700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f7f78030fa0 0x7f7f78033460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f7f800096a0 tx=0x7f7f80006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:15.826+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7f7c01ba10 con 0x7f7f8c072cb0 2024-01-11T05:03:16.030 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.030+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f7f8c10eb00 con 0x7f7f8c072cb0 2024-01-11T05:03:16.121 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:14.845329+0000 mon.smithi067 (mon.0) 65 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:16.121 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:15.021278+0000 mon.smithi067 (mon.0) 66 : audit [INF] from='client.? 172.21.15.67:0/2248337594' entity='client.admin' 2024-01-11T05:03:16.121 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:15.082373+0000 mon.smithi067 (mon.0) 67 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:16.121 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:15.808787+0000 mon.smithi067 (mon.0) 68 : audit [INF] from='mgr.14116 172.21.15.67:0/658640585' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.849+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f7f7c00f070 con 0x7f7f8c072cb0 2024-01-11T05:03:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.850+0000 7f7f73fff700 1 -- 172.21.15.67:0/790582510 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7f7c010360 con 0x7f7f8c072cb0 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f7f78030fa0 msgr2=0x7f7f78033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f7f78030fa0 0x7f7f78033460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f7f800096a0 tx=0x7f7f80006b40).stop 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 msgr2=0x7f7f8c10cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c10cf50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7f7c006800 tx=0x7f7f7c006ee0).stop 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 shutdown_connections 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f7f78030fa0 0x7f7f78033460 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 --2- 172.21.15.67:0/790582510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f8c072cb0 0x7f7f8c10cf50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.852+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 >> 172.21.15.67:0/790582510 conn(0x7f7f8c0f7780 msgr2=0x7f7f8c0f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.853+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 shutdown_connections 2024-01-11T05:03:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:16.853+0000 7f7f91e4f700 1 -- 172.21.15.67:0/790582510 wait complete. 2024-01-11T05:03:17.154 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:16 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:16.031309+0000 mon.smithi067 (mon.0) 69 : audit [INF] from='client.? 172.21.15.67:0/790582510' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-11T05:03:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.574+0000 7f3f43726700 1 Processor -- start 2024-01-11T05:03:17.577 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f43726700 1 -- start start 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f43726700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f43726700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0ff0d0 con 0x7f3f3c027670 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f414c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f414c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41834/0 (socket says 172.21.15.67:41834) 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f414c2700 1 -- 172.21.15.67:0/4022767251 learned_addr learned my addr 172.21.15.67:0/4022767251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f414c2700 1 -- 172.21.15.67:0/4022767251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0ff210 con 0x7f3f3c027670 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f414c2700 1 --2- 172.21.15.67:0/4022767251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0febe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f3f3800b5f0 tx=0x7f3f38006ee0).ready entity=mon.0 client_cookie=422369183d04f360 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f2bfff700 1 -- 172.21.15.67:0/4022767251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f3800d600 con 0x7f3f3c027670 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.575+0000 7f3f2bfff700 1 -- 172.21.15.67:0/4022767251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3f3800dbf0 con 0x7f3f3c027670 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.576+0000 7f3f2bfff700 1 -- 172.21.15.67:0/4022767251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f38011bf0 con 0x7f3f3c027670 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.576+0000 7f3f43726700 1 -- 172.21.15.67:0/4022767251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 msgr2=0x7f3f3c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:17.578 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.576+0000 7f3f43726700 1 --2- 172.21.15.67:0/4022767251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0febe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f3f3800b5f0 tx=0x7f3f38006ee0).stop 2024-01-11T05:03:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.577+0000 7f3f43726700 1 -- 172.21.15.67:0/4022767251 shutdown_connections 2024-01-11T05:03:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.577+0000 7f3f43726700 1 --2- 172.21.15.67:0/4022767251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0febe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.577+0000 7f3f43726700 1 -- 172.21.15.67:0/4022767251 >> 172.21.15.67:0/4022767251 conn(0x7f3f3c0f6680 msgr2=0x7f3f3c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.577+0000 7f3f43726700 1 -- 172.21.15.67:0/4022767251 shutdown_connections 2024-01-11T05:03:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.577+0000 7f3f43726700 1 -- 172.21.15.67:0/4022767251 wait complete. 2024-01-11T05:03:17.582 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f43726700 1 Processor -- start 2024-01-11T05:03:17.582 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f43726700 1 -- start start 2024-01-11T05:03:17.582 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f43726700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:17.582 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f43726700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0faae0 con 0x7f3f3c027670 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f414c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f414c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41850/0 (socket says 172.21.15.67:41850) 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f414c2700 1 -- 172.21.15.67:0/3737092351 learned_addr learned my addr 172.21.15.67:0/3737092351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f414c2700 1 -- 172.21.15.67:0/3737092351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3800b200 con 0x7f3f3c027670 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f414c2700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0fc660 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f3f38011de0 tx=0x7f3f38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f3801b410 con 0x7f3f3c027670 2024-01-11T05:03:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3f3801ba00 con 0x7f3f3c027670 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.578+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f3801a770 con 0x7f3f3c027670 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.579+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c0fac80 con 0x7f3f3c027670 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.579+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c0fb110 con 0x7f3f3c027670 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.579+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f3f3802b070 con 0x7f3f3c027670 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.580+0000 7f3f29ffb700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 0x7f3f24033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.580+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3f38042bd0 con 0x7f3f3c027670 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.580+0000 7f3f40cc1700 1 -- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 msgr2=0x7f3f24033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.580+0000 7f3f40cc1700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 0x7f3f24033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-11T05:03:17.584 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.580+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c0591e0 con 0x7f3f3c027670 2024-01-11T05:03:17.585 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.584+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3f38016020 con 0x7f3f3c027670 2024-01-11T05:03:17.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.780+0000 7f3f40cc1700 1 -- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 msgr2=0x7f3f24033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:17.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.780+0000 7f3f40cc1700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 0x7f3f24033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-11T05:03:17.788 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.787+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3f3c104bd0 con 0x7f3f3c027670 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.787+0000 7f3f29ffb700 1 -- 172.21.15.67:0/3737092351 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3f3801f070 con 0x7f3f3c027670 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "epoch": 9, 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "available": true, 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "active_name": "smithi067.srbzpc", 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-11T05:03:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 msgr2=0x7f3f24033500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 0x7f3f24033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 msgr2=0x7f3f3c0fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0fc660 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f3f38011de0 tx=0x7f3f38006ee0).stop 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 shutdown_connections 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f3f24031040 0x7f3f24033500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 --2- 172.21.15.67:0/3737092351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f3c027670 0x7f3f3c0fc660 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 >> 172.21.15.67:0/3737092351 conn(0x7f3f3c0f6680 msgr2=0x7f3f3c0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 shutdown_connections 2024-01-11T05:03:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:17.789+0000 7f3f43726700 1 -- 172.21.15.67:0/3737092351 wait complete. 2024-01-11T05:03:18.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:16.850733+0000 mon.smithi067 (mon.0) 70 : audit [INF] from='client.? 172.21.15.67:0/790582510' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-11T05:03:18.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:16.850809+0000 mon.smithi067 (mon.0) 71 : cluster [DBG] mgrmap e9: smithi067.srbzpc(active, since 17s) 2024-01-11T05:03:18.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:17.789039+0000 mon.smithi067 (mon.0) 72 : audit [DBG] from='client.? 172.21.15.67:0/3737092351' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-11T05:03:18.593 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for the mgr to restart... 2024-01-11T05:03:18.593 INFO:teuthology.orchestra.run.smithi067.stderr:Waiting for mgr epoch 9... 2024-01-11T05:03:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.932+0000 7f8fff312700 1 Processor -- start 2024-01-11T05:03:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.932+0000 7f8fff312700 1 -- start start 2024-01-11T05:03:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.933+0000 7f8fff312700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff80fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.933+0000 7f8fff312700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80fb7e0 con 0x7f8ff80faed0 2024-01-11T05:03:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.933+0000 7f8ffd0ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff80fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.933+0000 7f8ffd0ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff80fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41866/0 (socket says 172.21.15.67:41866) 2024-01-11T05:03:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.933+0000 7f8ffd0ae700 1 -- 172.21.15.67:0/1957725185 learned_addr learned my addr 172.21.15.67:0/1957725185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.933+0000 7f8ffd0ae700 1 -- 172.21.15.67:0/1957725185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff80fb920 con 0x7f8ff80faed0 2024-01-11T05:03:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8ffd0ae700 1 --2- 172.21.15.67:0/1957725185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff80fb2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8ff400b5f0 tx=0x7f8ff4006ee0).ready entity=mon.0 client_cookie=7e417be9815817bf server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fe7fff700 1 -- 172.21.15.67:0/1957725185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff400d600 con 0x7f8ff80faed0 2024-01-11T05:03:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fe7fff700 1 -- 172.21.15.67:0/1957725185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8ff400dbf0 con 0x7f8ff80faed0 2024-01-11T05:03:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fe7fff700 1 -- 172.21.15.67:0/1957725185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff4011bf0 con 0x7f8ff80faed0 2024-01-11T05:03:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fff312700 1 -- 172.21.15.67:0/1957725185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 msgr2=0x7f8ff80fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fff312700 1 --2- 172.21.15.67:0/1957725185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff80fb2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8ff400b5f0 tx=0x7f8ff4006ee0).stop 2024-01-11T05:03:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fff312700 1 -- 172.21.15.67:0/1957725185 shutdown_connections 2024-01-11T05:03:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fff312700 1 --2- 172.21.15.67:0/1957725185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff80fb2f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.934+0000 7f8fff312700 1 -- 172.21.15.67:0/1957725185 >> 172.21.15.67:0/1957725185 conn(0x7f8ff80f6680 msgr2=0x7f8ff80f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.935+0000 7f8fff312700 1 -- 172.21.15.67:0/1957725185 shutdown_connections 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.935+0000 7f8fff312700 1 -- 172.21.15.67:0/1957725185 wait complete. 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.935+0000 7f8fff312700 1 Processor -- start 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.935+0000 7f8fff312700 1 -- start start 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8fff312700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff810ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8fff312700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff810b270 con 0x7f8ff80faed0 2024-01-11T05:03:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8ffd0ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff810ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8ffd0ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff810ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41876/0 (socket says 172.21.15.67:41876) 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8ffd0ae700 1 -- 172.21.15.67:0/1294463470 learned_addr learned my addr 172.21.15.67:0/1294463470 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8ffd0ae700 1 -- 172.21.15.67:0/1294463470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff400b200 con 0x7f8ff80faed0 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8ffd0ae700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff810ad80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8ff4006800 tx=0x7f8ff4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff400dec0 con 0x7f8ff80faed0 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff810b470 con 0x7f8ff80faed0 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8ff401b8a0 con 0x7f8ff80faed0 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.936+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ff401a7b0 con 0x7f8ff80faed0 2024-01-11T05:03:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.937+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff810c080 con 0x7f8ff80faed0 2024-01-11T05:03:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.938+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f8ff402b070 con 0x7f8ff80faed0 2024-01-11T05:03:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.938+0000 7f8fe5ffb700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.938+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 --> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fe0033b70 con 0x7f8fe0030ff0 2024-01-11T05:03:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.938+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f8ff4042c00 con 0x7f8ff80faed0 2024-01-11T05:03:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.938+0000 7f8ffc8ad700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 msgr2=0x7f8fe00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:18.939+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-11T05:03:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:19.140+0000 7f8ffc8ad700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 msgr2=0x7f8fe00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:19.140+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-11T05:03:19.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:19.541+0000 7f8ffc8ad700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 msgr2=0x7f8fe00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:19.542 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:19.541+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-11T05:03:20.343 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:20.342+0000 7f8ffc8ad700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 msgr2=0x7f8fe00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:20.343 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:20.342+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-11T05:03:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:21.943+0000 7f8ffc8ad700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 msgr2=0x7f8fe00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3176394990 2024-01-11T05:03:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:21.943+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-11T05:03:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:22.263+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 10) v1 ==== 36958+0+0 (secure 0 0 0) 0x7f8ff400f070 con 0x7f8ff80faed0 2024-01-11T05:03:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:22.263+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 msgr2=0x7f8fe00334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-11T05:03:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:22.263+0000 7f8fe5ffb700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:03:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.264+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f8ff4044310 con 0x7f8ff80faed0 2024-01-11T05:03:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.264+0000 7f8fe5ffb700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8fe0034b40 0x7f8fe0036f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.264+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fe0033b70 con 0x7f8fe0034b40 2024-01-11T05:03:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.265+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8fe0034b40 0x7f8fe0036f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.265+0000 7f8ffc8ad700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8fe0034b40 0x7f8fe0036f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8fe8006fd0 tx=0x7f8fe8006d30).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.266+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f8fe0033b70 con 0x7f8fe0034b40 2024-01-11T05:03:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.272+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8ff810b900 con 0x7f8fe0034b40 2024-01-11T05:03:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.272+0000 7f8fe5ffb700 1 -- 172.21.15.67:0/1294463470 <== mgr.14156 v2:172.21.15.67:6800/1409268088 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8ff810b900 con 0x7f8fe0034b40 2024-01-11T05:03:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: { 2024-01-11T05:03:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-01-11T05:03:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: "initialized": true 2024-01-11T05:03:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: } 2024-01-11T05:03:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.272+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8fe0034b40 msgr2=0x7f8fe0036f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.272+0000 7f8fff312700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8fe0034b40 0x7f8fe0036f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8fe8006fd0 tx=0x7f8fe8006d30).stop 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.272+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 msgr2=0x7f8ff810ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.272+0000 7f8fff312700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff810ad80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8ff4006800 tx=0x7f8ff4006ee0).stop 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 shutdown_connections 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8fe0034b40 0x7f8fe0036f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ff80faed0 0x7f8ff810ad80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 --2- 172.21.15.67:0/1294463470 >> [v2:172.21.15.67:6800/3176394990,v1:172.21.15.67:6801/3176394990] conn(0x7f8fe0030ff0 0x7f8fe00334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 >> 172.21.15.67:0/1294463470 conn(0x7f8ff80f6680 msgr2=0x7f8ff80f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 shutdown_connections 2024-01-11T05:03:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:23.273+0000 7f8fff312700 1 -- 172.21.15.67:0/1294463470 wait complete. 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:22.210523+0000 mon.smithi067 (mon.0) 73 : cluster [INF] Active manager daemon smithi067.srbzpc restarted 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:22.210967+0000 mon.smithi067 (mon.0) 74 : cluster [INF] Activating manager daemon smithi067.srbzpc 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:22.215137+0000 mon.smithi067 (mon.0) 75 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:22.266164+0000 mon.smithi067 (mon.0) 76 : cluster [DBG] mgrmap e10: smithi067.srbzpc(active, starting, since 0.0553106s) 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.267208+0000 mon.smithi067 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.267846+0000 mon.smithi067 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mgr metadata", "who": "smithi067.srbzpc", "id": "smithi067.srbzpc"}]: dispatch 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.268533+0000 mon.smithi067 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-11T05:03:23.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.268702+0000 mon.smithi067 (mon.0) 80 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.268862+0000 mon.smithi067 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:22.283330+0000 mon.smithi067 (mon.0) 82 : cluster [INF] Manager daemon smithi067.srbzpc is now available 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.387940+0000 mon.smithi067 (mon.0) 83 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.390338+0000 mon.smithi067 (mon.0) 84 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.410441+0000 mon.smithi067 (mon.0) 85 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:22.412000+0000 mon.smithi067 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:03:23.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:23.023900+0000 mon.smithi067 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:mgr epoch 9 is available 2024-01-11T05:03:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:Generating a dashboard self-signed certificate... 2024-01-11T05:03:24.378 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:24 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:23.267663+0000 mgr.smithi067.srbzpc (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-11T05:03:24.378 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:24 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:23.268163+0000 mon.smithi067 (mon.0) 88 : cluster [DBG] mgrmap e11: smithi067.srbzpc(active, since 1.05731s) 2024-01-11T05:03:24.378 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:24 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:23.274211+0000 mgr.smithi067.srbzpc (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-11T05:03:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.776+0000 7f2fc1730700 1 Processor -- start 2024-01-11T05:03:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.776+0000 7f2fc1730700 1 -- start start 2024-01-11T05:03:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.776+0000 7f2fc1730700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc0fb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.776+0000 7f2fc1730700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc0fb680 con 0x7f2fbc0fad70 2024-01-11T05:03:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc0fb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc0fb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50374/0 (socket says 172.21.15.67:50374) 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbbfff700 1 -- 172.21.15.67:0/2423145754 learned_addr learned my addr 172.21.15.67:0/2423145754 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbbfff700 1 -- 172.21.15.67:0/2423145754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0fb7c0 con 0x7f2fbc0fad70 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbbfff700 1 --2- 172.21.15.67:0/2423145754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc0fb190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f2fac00b5f0 tx=0x7f2fac006ee0).ready entity=mon.0 client_cookie=28b25fc28a90e2c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbaffd700 1 -- 172.21.15.67:0/2423145754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac00d600 con 0x7f2fbc0fad70 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbaffd700 1 -- 172.21.15.67:0/2423145754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2fac00dbf0 con 0x7f2fbc0fad70 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.777+0000 7f2fbaffd700 1 -- 172.21.15.67:0/2423145754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac011bf0 con 0x7f2fbc0fad70 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 -- 172.21.15.67:0/2423145754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 msgr2=0x7f2fbc0fb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 --2- 172.21.15.67:0/2423145754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc0fb190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f2fac00b5f0 tx=0x7f2fac006ee0).stop 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 -- 172.21.15.67:0/2423145754 shutdown_connections 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 --2- 172.21.15.67:0/2423145754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc0fb190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 -- 172.21.15.67:0/2423145754 >> 172.21.15.67:0/2423145754 conn(0x7f2fbc0f64d0 msgr2=0x7f2fbc0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 -- 172.21.15.67:0/2423145754 shutdown_connections 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.778+0000 7f2fc1730700 1 -- 172.21.15.67:0/2423145754 wait complete. 2024-01-11T05:03:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fc1730700 1 Processor -- start 2024-01-11T05:03:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fc1730700 1 -- start start 2024-01-11T05:03:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fc1730700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc10d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fc1730700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc06ad80 con 0x7f2fbc0fad70 2024-01-11T05:03:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc10d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc10d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50388/0 (socket says 172.21.15.67:50388) 2024-01-11T05:03:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.779+0000 7f2fbbfff700 1 -- 172.21.15.67:0/1284672435 learned_addr learned my addr 172.21.15.67:0/1284672435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fbbfff700 1 -- 172.21.15.67:0/1284672435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac00b200 con 0x7f2fbc0fad70 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fbbfff700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc10d830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2fac011e30 tx=0x7f2fac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac01b410 con 0x7f2fbc0fad70 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2fac01ba00 con 0x7f2fbc0fad70 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc06af80 con 0x7f2fbc0fad70 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac01a750 con 0x7f2fbc0fad70 2024-01-11T05:03:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.780+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc10dee0 con 0x7f2fbc0fad70 2024-01-11T05:03:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.781+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f2fac02b070 con 0x7f2fbc0fad70 2024-01-11T05:03:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.781+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc045bb0 con 0x7f2fbc0fad70 2024-01-11T05:03:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.781+0000 7f2fb8ff9700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fa8030f60 0x7f2fa8033420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.781+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2fac042b00 con 0x7f2fbc0fad70 2024-01-11T05:03:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.781+0000 7f2fbb7fe700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fa8030f60 0x7f2fa8033420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.781+0000 7f2fbb7fe700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fa8030f60 0x7f2fa8033420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2fb00096a0 tx=0x7f2fb0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.785+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2fac02b8f0 con 0x7f2fbc0fad70 2024-01-11T05:03:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.953+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fbc057ff0 con 0x7f2fa8030f60 2024-01-11T05:03:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:24.995+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f2fac00f070 con 0x7f2fbc0fad70 2024-01-11T05:03:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.017+0000 7f2fb8ff9700 1 -- 172.21.15.67:0/1284672435 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f2fbc057ff0 con 0x7f2fa8030f60 2024-01-11T05:03:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fa8030f60 msgr2=0x7f2fa8033420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fa8030f60 0x7f2fa8033420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2fb00096a0 tx=0x7f2fb0006b40).stop 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 msgr2=0x7f2fbc10d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc10d830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2fac011e30 tx=0x7f2fac006ee0).stop 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 shutdown_connections 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fa8030f60 0x7f2fa8033420 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 --2- 172.21.15.67:0/1284672435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fbc0fad70 0x7f2fbc10d830 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.019+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 >> 172.21.15.67:0/1284672435 conn(0x7f2fbc0f64d0 msgr2=0x7f2fbc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.020+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 shutdown_connections 2024-01-11T05:03:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.020+0000 7f2fc1730700 1 -- 172.21.15.67:0/1284672435 wait complete. 2024-01-11T05:03:25.393 INFO:teuthology.orchestra.run.smithi067.stderr:Creating initial admin user... 2024-01-11T05:03:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.744+0000 7f3a90815700 1 Processor -- start 2024-01-11T05:03:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a90815700 1 -- start start 2024-01-11T05:03:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a90815700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c070100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a90815700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c072020 con 0x7f3a8c071c00 2024-01-11T05:03:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a8a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a8a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50398/0 (socket says 172.21.15.67:50398) 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a8a59c700 1 -- 172.21.15.67:0/3045200821 learned_addr learned my addr 172.21.15.67:0/3045200821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a8a59c700 1 -- 172.21.15.67:0/3045200821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0705f0 con 0x7f3a8c071c00 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.745+0000 7f3a8a59c700 1 --2- 172.21.15.67:0/3045200821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c070100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f3a7c00b5f0 tx=0x7f3a7c006ee0).ready entity=mon.0 client_cookie=5dfb89c06234ba25 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a8959a700 1 -- 172.21.15.67:0/3045200821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a7c00d600 con 0x7f3a8c071c00 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a8959a700 1 -- 172.21.15.67:0/3045200821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3a7c00dbf0 con 0x7f3a8c071c00 2024-01-11T05:03:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a8959a700 1 -- 172.21.15.67:0/3045200821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a7c011bf0 con 0x7f3a8c071c00 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a90815700 1 -- 172.21.15.67:0/3045200821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 msgr2=0x7f3a8c070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a90815700 1 --2- 172.21.15.67:0/3045200821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c070100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f3a7c00b5f0 tx=0x7f3a7c006ee0).stop 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a90815700 1 -- 172.21.15.67:0/3045200821 shutdown_connections 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a90815700 1 --2- 172.21.15.67:0/3045200821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c070100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.746+0000 7f3a90815700 1 -- 172.21.15.67:0/3045200821 >> 172.21.15.67:0/3045200821 conn(0x7f3a8c0f7770 msgr2=0x7f3a8c0f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a90815700 1 -- 172.21.15.67:0/3045200821 shutdown_connections 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a90815700 1 -- 172.21.15.67:0/3045200821 wait complete. 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a90815700 1 Processor -- start 2024-01-11T05:03:25.750 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a90815700 1 -- start start 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a90815700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a90815700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c10d4d0 con 0x7f3a8c071c00 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a8a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.747+0000 7f3a8a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c10cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50406/0 (socket says 172.21.15.67:50406) 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a8a59c700 1 -- 172.21.15.67:0/460235428 learned_addr learned my addr 172.21.15.67:0/460235428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a8a59c700 1 -- 172.21.15.67:0/460235428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c00b200 con 0x7f3a8c071c00 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a8a59c700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c10cfe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3a7c006800 tx=0x7f3a7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a7c00dec0 con 0x7f3a8c071c00 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3a7c01b8a0 con 0x7f3a8c071c00 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c10d6d0 con 0x7f3a8c071c00 2024-01-11T05:03:25.751 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a7c01a700 con 0x7f3a8c071c00 2024-01-11T05:03:25.752 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.748+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c10db60 con 0x7f3a8c071c00 2024-01-11T05:03:25.752 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.749+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f3a7c02b070 con 0x7f3a8c071c00 2024-01-11T05:03:25.752 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.749+0000 7f3a7affd700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3a70030ff0 0x7f3a700334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:25.752 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.749+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3a7c042bf0 con 0x7f3a8c071c00 2024-01-11T05:03:25.753 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.749+0000 7f3a89d9b700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3a70030ff0 0x7f3a700334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:25.753 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.749+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8c045bb0 con 0x7f3a8c071c00 2024-01-11T05:03:25.753 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.750+0000 7f3a89d9b700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3a70030ff0 0x7f3a700334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3a800096a0 tx=0x7f3a80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:25.755 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.754+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3a7c01f080 con 0x7f3a8c071c00 2024-01-11T05:03:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:25.930+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3a8c1021e0 con 0x7f3a70030ff0 2024-01-11T05:03:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.177+0000 7f3a7affd700 1 -- 172.21.15.67:0/460235428 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3a800092a0 con 0x7f3a70030ff0 2024-01-11T05:03:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$KDbW0RmJGJElr05aLZhgO.7jOowtUOUSvCX4nALthzbHIa/VaJuD6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1704949406, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-11T05:03:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3a70030ff0 msgr2=0x7f3a700334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3a70030ff0 0x7f3a700334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3a800096a0 tx=0x7f3a80006b40).stop 2024-01-11T05:03:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 msgr2=0x7f3a8c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c10cfe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3a7c006800 tx=0x7f3a7c006ee0).stop 2024-01-11T05:03:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 shutdown_connections 2024-01-11T05:03:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3a70030ff0 0x7f3a700334b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 --2- 172.21.15.67:0/460235428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8c071c00 0x7f3a8c10cfe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 >> 172.21.15.67:0/460235428 conn(0x7f3a8c0f7770 msgr2=0x7f3a8c0f9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 shutdown_connections 2024-01-11T05:03:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.179+0000 7f3a90815700 1 -- 172.21.15.67:0/460235428 wait complete. 2024-01-11T05:03:26.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:24.955437+0000 mgr.smithi067.srbzpc (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:26.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:24.998501+0000 mon.smithi067 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi067.srbzpc(active, since 2s) 2024-01-11T05:03:26.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:25.016983+0000 mon.smithi067 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:26.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:25.018795+0000 mon.smithi067 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:Fetching dashboard port number... 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.910+0000 7fe2ed9f0700 1 Processor -- start 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.910+0000 7fe2ed9f0700 1 -- start start 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2ed9f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e80fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2ed9f0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80fb7e0 con 0x7fe2e80faed0 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e80fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e80fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50418/0 (socket says 172.21.15.67:50418) 2024-01-11T05:03:26.914 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e6ffd700 1 -- 172.21.15.67:0/3729055325 learned_addr learned my addr 172.21.15.67:0/3729055325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e6ffd700 1 -- 172.21.15.67:0/3729055325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e80fb920 con 0x7fe2e80faed0 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e6ffd700 1 --2- 172.21.15.67:0/3729055325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e80fb2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe2dc00b870 tx=0x7fe2dc006f70).ready entity=mon.0 client_cookie=6ec03c0de24e3fe5 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e5ffb700 1 -- 172.21.15.67:0/3729055325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc00da20 con 0x7fe2e80faed0 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e5ffb700 1 -- 172.21.15.67:0/3729055325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2dc010470 con 0x7fe2e80faed0 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.911+0000 7fe2e5ffb700 1 -- 172.21.15.67:0/3729055325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc00f5b0 con 0x7fe2e80faed0 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/3729055325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 msgr2=0x7fe2e80fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 --2- 172.21.15.67:0/3729055325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e80fb2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe2dc00b870 tx=0x7fe2dc006f70).stop 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/3729055325 shutdown_connections 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 --2- 172.21.15.67:0/3729055325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e80fb2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/3729055325 >> 172.21.15.67:0/3729055325 conn(0x7fe2e80f6680 msgr2=0x7fe2e80f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/3729055325 shutdown_connections 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.912+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/3729055325 wait complete. 2024-01-11T05:03:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2ed9f0700 1 Processor -- start 2024-01-11T05:03:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2ed9f0700 1 -- start start 2024-01-11T05:03:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2ed9f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e810cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2ed9f0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e810d470 con 0x7fe2e80faed0 2024-01-11T05:03:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e810cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e810cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50432/0 (socket says 172.21.15.67:50432) 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2e6ffd700 1 -- 172.21.15.67:0/2282589104 learned_addr learned my addr 172.21.15.67:0/2282589104 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2e6ffd700 1 -- 172.21.15.67:0/2282589104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc00b480 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.913+0000 7fe2e6ffd700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e810cf80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe2dc00b840 tx=0x7fe2dc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.914+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc00f870 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.914+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2dc00fe60 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.914+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e810d670 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.914+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc019c60 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.914+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e810db00 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.915+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fe2dc02a070 con 0x7fe2e80faed0 2024-01-11T05:03:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.915+0000 7fe2cf7fe700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe2d4031040 0x7fe2d4033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.915+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2dc042ee0 con 0x7fe2e80faed0 2024-01-11T05:03:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.915+0000 7fe2e67fc700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe2d4031040 0x7fe2d4033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.915+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e8045bb0 con 0x7fe2e80faed0 2024-01-11T05:03:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.915+0000 7fe2e67fc700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe2d4031040 0x7fe2d4033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe2d80096a0 tx=0x7fe2d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:26.919+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe2dc00edc0 con 0x7fe2e80faed0 2024-01-11T05:03:27.082 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.080+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe2e80fcbd0 con 0x7fe2e80faed0 2024-01-11T05:03:27.082 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.080+0000 7fe2cf7fe700 1 -- 172.21.15.67:0/2282589104 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe2dc00e360 con 0x7fe2e80faed0 2024-01-11T05:03:27.082 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 8443 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe2d4031040 msgr2=0x7fe2d4033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe2d4031040 0x7fe2d4033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe2d80096a0 tx=0x7fe2d8006b40).stop 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 msgr2=0x7fe2e810cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e810cf80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe2dc00b840 tx=0x7fe2dc006f70).stop 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 shutdown_connections 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe2d4031040 0x7fe2d4033500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 --2- 172.21.15.67:0/2282589104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e80faed0 0x7fe2e810cf80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 >> 172.21.15.67:0/2282589104 conn(0x7fe2e80f6680 msgr2=0x7fe2e80ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 shutdown_connections 2024-01-11T05:03:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.082+0000 7fe2ed9f0700 1 -- 172.21.15.67:0/2282589104 wait complete. 2024-01-11T05:03:27.361 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:25.932623+0000 mgr.smithi067.srbzpc (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:27.361 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:26.179436+0000 mon.smithi067 (mon.0) 92 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:27.361 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:27.082466+0000 mon.smithi067 (mon.0) 93 : audit [DBG] from='client.? 172.21.15.67:0/2282589104' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-11T05:03:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: disabled 2024-01-11T05:03:27.467 INFO:teuthology.orchestra.run.smithi067.stderr:systemctl: inactive 2024-01-11T05:03:27.468 INFO:teuthology.orchestra.run.smithi067.stderr:firewalld.service is not enabled 2024-01-11T05:03:27.468 INFO:teuthology.orchestra.run.smithi067.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-11T05:03:27.470 INFO:teuthology.orchestra.run.smithi067.stderr:Ceph Dashboard is now available at: 2024-01-11T05:03:27.470 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:27.470 INFO:teuthology.orchestra.run.smithi067.stderr: URL: https://smithi067.front.sepia.ceph.com:8443/ 2024-01-11T05:03:27.471 INFO:teuthology.orchestra.run.smithi067.stderr: User: admin 2024-01-11T05:03:27.471 INFO:teuthology.orchestra.run.smithi067.stderr: Password: 34nlhsrm0v 2024-01-11T05:03:27.471 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:Enabling autotune for osd_memory_target 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.799+0000 7f9e36e1d700 1 Processor -- start 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.799+0000 7f9e36e1d700 1 -- start start 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e36e1d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e300fda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e36e1d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e300fdf70 con 0x7f9e300fd660 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e34bb9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e300fda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e34bb9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e300fda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50448/0 (socket says 172.21.15.67:50448) 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e34bb9700 1 -- 172.21.15.67:0/4168860844 learned_addr learned my addr 172.21.15.67:0/4168860844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e34bb9700 1 -- 172.21.15.67:0/4168860844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e300fe0b0 con 0x7f9e300fd660 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e34bb9700 1 --2- 172.21.15.67:0/4168860844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e300fda80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f9e2400b5f0 tx=0x7f9e24006ee0).ready entity=mon.0 client_cookie=e708b9503f5d403e server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e2f7fe700 1 -- 172.21.15.67:0/4168860844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2400d600 con 0x7f9e300fd660 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e2f7fe700 1 -- 172.21.15.67:0/4168860844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9e2400dbf0 con 0x7f9e300fd660 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.800+0000 7f9e2f7fe700 1 -- 172.21.15.67:0/4168860844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e24011bf0 con 0x7f9e300fd660 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 -- 172.21.15.67:0/4168860844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 msgr2=0x7f9e300fda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 --2- 172.21.15.67:0/4168860844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e300fda80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f9e2400b5f0 tx=0x7f9e24006ee0).stop 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 -- 172.21.15.67:0/4168860844 shutdown_connections 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 --2- 172.21.15.67:0/4168860844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e300fda80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 -- 172.21.15.67:0/4168860844 >> 172.21.15.67:0/4168860844 conn(0x7f9e300f5e20 msgr2=0x7f9e300f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 -- 172.21.15.67:0/4168860844 shutdown_connections 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.801+0000 7f9e36e1d700 1 -- 172.21.15.67:0/4168860844 wait complete. 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e36e1d700 1 Processor -- start 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e36e1d700 1 -- start start 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e36e1d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e3010ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e36e1d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010b290 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e34bb9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e3010ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e34bb9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e3010ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50464/0 (socket says 172.21.15.67:50464) 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.802+0000 7f9e34bb9700 1 -- 172.21.15.67:0/273144883 learned_addr learned my addr 172.21.15.67:0/273144883 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e34bb9700 1 -- 172.21.15.67:0/273144883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2400b200 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e34bb9700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e3010ada0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9e24011de0 tx=0x7f9e24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2401b410 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3010b490 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9e2401ba00 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2401a770 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.803+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3010c0d0 con 0x7f9e300fd660 2024-01-11T05:03:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.804+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f9e2402b070 con 0x7f9e300fd660 2024-01-11T05:03:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.804+0000 7f9e2d7fa700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e14030ff0 0x7f9e140334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.804+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9e24042e00 con 0x7f9e300fd660 2024-01-11T05:03:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.804+0000 7f9e2ffff700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e14030ff0 0x7f9e140334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.804+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3010b920 con 0x7f9e300fd660 2024-01-11T05:03:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.804+0000 7f9e2ffff700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e14030ff0 0x7f9e140334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9e1c0096a0 tx=0x7f9e1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.808+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9e24016080 con 0x7f9e300fd660 2024-01-11T05:03:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.967+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f9e300591e0 con 0x7f9e300fd660 2024-01-11T05:03:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.968+0000 7f9e2d7fa700 1 -- 172.21.15.67:0/273144883 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v10)=0 v10) v1 ==== 127+0+0 (secure 0 0 0) 0x7f9e2401f020 con 0x7f9e300fd660 2024-01-11T05:03:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e14030ff0 msgr2=0x7f9e140334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e14030ff0 0x7f9e140334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9e1c0096a0 tx=0x7f9e1c006b40).stop 2024-01-11T05:03:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 msgr2=0x7f9e3010ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e3010ada0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9e24011de0 tx=0x7f9e24006ee0).stop 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 shutdown_connections 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e14030ff0 0x7f9e140334b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 --2- 172.21.15.67:0/273144883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e300fd660 0x7f9e3010ada0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 >> 172.21.15.67:0/273144883 conn(0x7f9e300f5e20 msgr2=0x7f9e300ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 shutdown_connections 2024-01-11T05:03:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:27.969+0000 7f9e36e1d700 1 -- 172.21.15.67:0/273144883 wait complete. 2024-01-11T05:03:28.679 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.675+0000 7f6d0aba6700 1 Processor -- start 2024-01-11T05:03:28.679 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.675+0000 7f6d0aba6700 1 -- start start 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d0aba6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d0aba6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040ff0d0 con 0x7f6d0406ade0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d08942700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d08942700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50470/0 (socket says 172.21.15.67:50470) 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d08942700 1 -- 172.21.15.67:0/2949228255 learned_addr learned my addr 172.21.15.67:0/2949228255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d08942700 1 -- 172.21.15.67:0/2949228255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040ff210 con 0x7f6d0406ade0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d08942700 1 --2- 172.21.15.67:0/2949228255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040febe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6cf800b5f0 tx=0x7f6cf8006ee0).ready entity=mon.0 client_cookie=5e473179aee92fd server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d037fe700 1 -- 172.21.15.67:0/2949228255 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf800d600 con 0x7f6d0406ade0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d037fe700 1 -- 172.21.15.67:0/2949228255 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6cf800dbf0 con 0x7f6d0406ade0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.676+0000 7f6d037fe700 1 -- 172.21.15.67:0/2949228255 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf8011bf0 con 0x7f6d0406ade0 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 -- 172.21.15.67:0/2949228255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 msgr2=0x7f6d040febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 --2- 172.21.15.67:0/2949228255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040febe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6cf800b5f0 tx=0x7f6cf8006ee0).stop 2024-01-11T05:03:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 -- 172.21.15.67:0/2949228255 shutdown_connections 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 --2- 172.21.15.67:0/2949228255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040febe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 -- 172.21.15.67:0/2949228255 >> 172.21.15.67:0/2949228255 conn(0x7f6d040f6680 msgr2=0x7f6d040f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 -- 172.21.15.67:0/2949228255 shutdown_connections 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.677+0000 7f6d0aba6700 1 -- 172.21.15.67:0/2949228255 wait complete. 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d0aba6700 1 Processor -- start 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d0aba6700 1 -- start start 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d0aba6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d0aba6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040faae0 con 0x7f6d0406ade0 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d08942700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d08942700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50484/0 (socket says 172.21.15.67:50484) 2024-01-11T05:03:28.681 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d08942700 1 -- 172.21.15.67:0/718891567 learned_addr learned my addr 172.21.15.67:0/718891567 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:28.682 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d08942700 1 -- 172.21.15.67:0/718891567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf800b200 con 0x7f6d0406ade0 2024-01-11T05:03:28.682 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.678+0000 7f6d08942700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040fc660 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6cf8011de0 tx=0x7f6cf8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:28.682 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.679+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf801b410 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.679+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6cf801ba00 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.679+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d040fac80 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.679+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf801a770 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.679+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d040fb110 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.679+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6cf802b070 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.680+0000 7f6d017fa700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6ce8031090 0x7f6ce8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.680+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6cf8042c90 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.680+0000 7f6d03fff700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6ce8031090 0x7f6ce8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.680+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d040591e0 con 0x7f6d0406ade0 2024-01-11T05:03:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.680+0000 7f6d03fff700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6ce8031090 0x7f6ce8033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f6cf00096a0 tx=0x7f6cf0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:28.685 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.684+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6cf8016070 con 0x7f6d0406ade0 2024-01-11T05:03:28.934 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.932+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f6d040ffed0 con 0x7f6d0406ade0 2024-01-11T05:03:28.935 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.934+0000 7f6d017fa700 1 -- 172.21.15.67:0/718891567 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v31)=0 set mgr/dashboard/cluster/status v31) v1 ==== 153+0+0 (secure 0 0 0) 0x7f6cf801b570 con 0x7f6d0406ade0 2024-01-11T05:03:28.938 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6ce8031090 msgr2=0x7f6ce8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6ce8031090 0x7f6ce8033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f6cf00096a0 tx=0x7f6cf0006b40).stop 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 msgr2=0x7f6d040fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040fc660 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6cf8011de0 tx=0x7f6cf8006ee0).stop 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 shutdown_connections 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6ce8031090 0x7f6ce8033550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.936+0000 7f6d0aba6700 1 --2- 172.21.15.67:0/718891567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0406ade0 0x7f6d040fc660 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.937+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 >> 172.21.15.67:0/718891567 conn(0x7f6d040f6680 msgr2=0x7f6d040f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.937+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 shutdown_connections 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: 2024-01-11T05:03:28.937+0000 7f6d0aba6700 1 -- 172.21.15.67:0/718891567 wait complete. 2024-01-11T05:03:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr:Or, if you are only running a single cluster on this host: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: ceph telemetry on 2024-01-11T05:03:29.334 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:For more information see: 2024-01-11T05:03:29.335 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.335 INFO:teuthology.orchestra.run.smithi067.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-01-11T05:03:29.335 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:03:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:Bootstrap complete. 2024-01-11T05:03:29.369 INFO:tasks.cephadm:Fetching config... 2024-01-11T05:03:29.369 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:03:29.369 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-11T05:03:29.427 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-11T05:03:29.428 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:03:29.428 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-11T05:03:29.488 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-11T05:03:29.488 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:03:29.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/mon.smithi067/keyring of=/dev/stdout 2024-01-11T05:03:29.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:28.277338+0000 mon.smithi067 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi067.srbzpc(active, since 6s) 2024-01-11T05:03:29.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:28.936553+0000 mon.smithi067 (mon.0) 95 : audit [INF] from='client.? 172.21.15.67:0/718891567' entity='client.admin' 2024-01-11T05:03:29.558 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-11T05:03:29.558 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-01-11T05:03:29.558 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-11T05:03:29.617 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-11T05:03:29.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBpB38VCx/OpP3BJyxeQAwHkbLjge+TEnE+IOD2KjVixmHcis50d0HxHpewifyRY/gFw87FQO4mf/jdBk7mqXqopLqt8KDQFzplBwGq5BT7SjOgROzZUPHU0UYPtfOfZ0EOZguImUL6JnwwZMDmbCwbfchmwBfNzX/jgDrXoivcEGfpkfqpDvnYO5LxP+JyMC6MEcG/RuE0CRp0utUVoJ+5k/XCnGAJDeI2tAgqDlcp4vcOQl8QhFKrlhSfhM+p5oyegmpJa8vA9DxoNfrJhOHEaLR0DVOu9Zt6i8LCvDqt5JDJhnfLwpiqMtMeIM8fd5dwtogJOAdcXukg6lYgxpm/HsAg87NfZY9bu93AEGfx1+wLDeJWZRCj6k3yrcra9YkT0MeWDfIxoKeoUcFyrJeZ1uIMBAtHxkK38AT/0LZQot/KeeXROSZBVr3fDYI7IOiu5gCgs5n1CLVZvwUC/Sdmo9XZid7YfYhhTJPqv1rQ3uPLBf4WN7Uq6vFTlRpjxc= ceph-89773ea0-b03e-11ee-95ab-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-11T05:03:29.696 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBpB38VCx/OpP3BJyxeQAwHkbLjge+TEnE+IOD2KjVixmHcis50d0HxHpewifyRY/gFw87FQO4mf/jdBk7mqXqopLqt8KDQFzplBwGq5BT7SjOgROzZUPHU0UYPtfOfZ0EOZguImUL6JnwwZMDmbCwbfchmwBfNzX/jgDrXoivcEGfpkfqpDvnYO5LxP+JyMC6MEcG/RuE0CRp0utUVoJ+5k/XCnGAJDeI2tAgqDlcp4vcOQl8QhFKrlhSfhM+p5oyegmpJa8vA9DxoNfrJhOHEaLR0DVOu9Zt6i8LCvDqt5JDJhnfLwpiqMtMeIM8fd5dwtogJOAdcXukg6lYgxpm/HsAg87NfZY9bu93AEGfx1+wLDeJWZRCj6k3yrcra9YkT0MeWDfIxoKeoUcFyrJeZ1uIMBAtHxkK38AT/0LZQot/KeeXROSZBVr3fDYI7IOiu5gCgs5n1CLVZvwUC/Sdmo9XZid7YfYhhTJPqv1rQ3uPLBf4WN7Uq6vFTlRpjxc= ceph-89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:03:29.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBpB38VCx/OpP3BJyxeQAwHkbLjge+TEnE+IOD2KjVixmHcis50d0HxHpewifyRY/gFw87FQO4mf/jdBk7mqXqopLqt8KDQFzplBwGq5BT7SjOgROzZUPHU0UYPtfOfZ0EOZguImUL6JnwwZMDmbCwbfchmwBfNzX/jgDrXoivcEGfpkfqpDvnYO5LxP+JyMC6MEcG/RuE0CRp0utUVoJ+5k/XCnGAJDeI2tAgqDlcp4vcOQl8QhFKrlhSfhM+p5oyegmpJa8vA9DxoNfrJhOHEaLR0DVOu9Zt6i8LCvDqt5JDJhnfLwpiqMtMeIM8fd5dwtogJOAdcXukg6lYgxpm/HsAg87NfZY9bu93AEGfx1+wLDeJWZRCj6k3yrcra9YkT0MeWDfIxoKeoUcFyrJeZ1uIMBAtHxkK38AT/0LZQot/KeeXROSZBVr3fDYI7IOiu5gCgs5n1CLVZvwUC/Sdmo9XZid7YfYhhTJPqv1rQ3uPLBf4WN7Uq6vFTlRpjxc= ceph-89773ea0-b03e-11ee-95ab-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-11T05:03:29.752 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBpB38VCx/OpP3BJyxeQAwHkbLjge+TEnE+IOD2KjVixmHcis50d0HxHpewifyRY/gFw87FQO4mf/jdBk7mqXqopLqt8KDQFzplBwGq5BT7SjOgROzZUPHU0UYPtfOfZ0EOZguImUL6JnwwZMDmbCwbfchmwBfNzX/jgDrXoivcEGfpkfqpDvnYO5LxP+JyMC6MEcG/RuE0CRp0utUVoJ+5k/XCnGAJDeI2tAgqDlcp4vcOQl8QhFKrlhSfhM+p5oyegmpJa8vA9DxoNfrJhOHEaLR0DVOu9Zt6i8LCvDqt5JDJhnfLwpiqMtMeIM8fd5dwtogJOAdcXukg6lYgxpm/HsAg87NfZY9bu93AEGfx1+wLDeJWZRCj6k3yrcra9YkT0MeWDfIxoKeoUcFyrJeZ1uIMBAtHxkK38AT/0LZQot/KeeXROSZBVr3fDYI7IOiu5gCgs5n1CLVZvwUC/Sdmo9XZid7YfYhhTJPqv1rQ3uPLBf4WN7Uq6vFTlRpjxc= ceph-89773ea0-b03e-11ee-95ab-87774f69a715 2024-01-11T05:03:29.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 -- 172.21.15.67:0/3902264676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 msgr2=0x7fd8900f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 --2- 172.21.15.67:0/3902264676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd8900f69b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd880009a00 tx=0x7fd880004cb0).stop 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 -- 172.21.15.67:0/3902264676 shutdown_connections 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 --2- 172.21.15.67:0/3902264676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd8900f69b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 -- 172.21.15.67:0/3902264676 >> 172.21.15.67:0/3902264676 conn(0x7fd8900f1cd0 msgr2=0x7fd8900f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 -- 172.21.15.67:0/3902264676 shutdown_connections 2024-01-11T05:03:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 -- 172.21.15.67:0/3902264676 wait complete. 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 Processor -- start 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.092+0000 7fd894a06700 1 -- start start 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd894a06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd890106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd894a06700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890106c60 con 0x7fd8900f6590 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd88e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd890106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd88e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd890106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50498/0 (socket says 172.21.15.67:50498) 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd88e59c700 1 -- 172.21.15.67:0/1095301908 learned_addr learned my addr 172.21.15.67:0/1095301908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd88e59c700 1 -- 172.21.15.67:0/1095301908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880009640 con 0x7fd8900f6590 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd88e59c700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd890106770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd880009610 tx=0x7fd880004cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8800092e0 con 0x7fd8900f6590 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd890106e60 con 0x7fd8900f6590 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd880007d50 con 0x7fd8900f6590 2024-01-11T05:03:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd880003de0 con 0x7fd8900f6590 2024-01-11T05:03:31.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.093+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8901072f0 con 0x7fd8900f6590 2024-01-11T05:03:31.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.094+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd88001f440 con 0x7fd8900f6590 2024-01-11T05:03:31.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.094+0000 7fd886ffd700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd87c031090 0x7fd87c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.094+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd880042d90 con 0x7fd8900f6590 2024-01-11T05:03:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.094+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd890045bb0 con 0x7fd8900f6590 2024-01-11T05:03:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.094+0000 7fd88dd9b700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd87c031090 0x7fd87c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.095+0000 7fd88dd9b700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd87c031090 0x7fd87c033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd8780096a0 tx=0x7fd878006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:31.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.099+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd880003450 con 0x7fd8900f6590 2024-01-11T05:03:31.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.269+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd8900fc080 con 0x7fd8900f6590 2024-01-11T05:03:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.274+0000 7fd886ffd700 1 -- 172.21.15.67:0/1095301908 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd880010310 con 0x7fd8900f6590 2024-01-11T05:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd87c031090 msgr2=0x7fd87c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd87c031090 0x7fd87c033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd8780096a0 tx=0x7fd878006b40).stop 2024-01-11T05:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 msgr2=0x7fd890106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd890106770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd880009610 tx=0x7fd880004cb0).stop 2024-01-11T05:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 shutdown_connections 2024-01-11T05:03:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd87c031090 0x7fd87c033550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 --2- 172.21.15.67:0/1095301908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900f6590 0x7fd890106770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 >> 172.21.15.67:0/1095301908 conn(0x7fd8900f1cd0 msgr2=0x7fd8900f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 shutdown_connections 2024-01-11T05:03:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:31.276+0000 7fd894a06700 1 -- 172.21.15.67:0/1095301908 wait complete. 2024-01-11T05:03:31.687 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-11T05:03:31.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-11T05:03:32.399 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:32 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:31.274723+0000 mon.smithi067 (mon.0) 96 : audit [INF] from='client.? 172.21.15.67:0/1095301908' entity='client.admin' 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/3928048824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 msgr2=0x7f3fd80f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 --2- 172.21.15.67:0/3928048824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80f5710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3fc8009fb0 tx=0x7f3fc8009b30).stop 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/3928048824 shutdown_connections 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 --2- 172.21.15.67:0/3928048824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80f5710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/3928048824 >> 172.21.15.67:0/3928048824 conn(0x7f3fd80f0a70 msgr2=0x7f3fd80f2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/3928048824 shutdown_connections 2024-01-11T05:03:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.044+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/3928048824 wait complete. 2024-01-11T05:03:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fdd9e8700 1 Processor -- start 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fdd9e8700 1 -- start start 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fdd9e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fdd9e8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd80fb750 con 0x7f3fd80f52f0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fd6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fd6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80fd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45174/0 (socket says 172.21.15.67:45174) 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.045+0000 7f3fd6ffd700 1 -- 172.21.15.67:0/1980541576 learned_addr learned my addr 172.21.15.67:0/1980541576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fd6ffd700 1 -- 172.21.15.67:0/1980541576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8005040 con 0x7f3fd80f52f0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fd6ffd700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80fd2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f3fc80089f0 tx=0x7f3fc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fc8009320 con 0x7f3fd80f52f0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fc8013ec0 con 0x7f3fd80f52f0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd80fb8f0 con 0x7f3fd80f52f0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fc8014d30 con 0x7f3fd80f52f0 2024-01-11T05:03:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.046+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd80fbd80 con 0x7f3fd80f52f0 2024-01-11T05:03:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.047+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd80591e0 con 0x7f3fd80f52f0 2024-01-11T05:03:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.047+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3fc80095f0 con 0x7f3fd80f52f0 2024-01-11T05:03:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.048+0000 7f3fcf7fe700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc40310e0 0x7f3fc40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.048+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fc8042510 con 0x7f3fd80f52f0 2024-01-11T05:03:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.049+0000 7f3fd67fc700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc40310e0 0x7f3fc40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.049+0000 7f3fd67fc700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc40310e0 0x7f3fc40335a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f3fc00096a0 tx=0x7f3fc0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.053+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fc800f940 con 0x7f3fd80f52f0 2024-01-11T05:03:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.224+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fd80fc5d0 con 0x7f3fc40310e0 2024-01-11T05:03:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.227+0000 7f3fcf7fe700 1 -- 172.21.15.67:0/1980541576 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3fd80fc5d0 con 0x7f3fc40310e0 2024-01-11T05:03:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc40310e0 msgr2=0x7f3fc40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc40310e0 0x7f3fc40335a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f3fc00096a0 tx=0x7f3fc0006b40).stop 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 msgr2=0x7f3fd80fd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80fd2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f3fc80089f0 tx=0x7f3fc800b040).stop 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 shutdown_connections 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc40310e0 0x7f3fc40335a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 --2- 172.21.15.67:0/1980541576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd80f52f0 0x7f3fd80fd2d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 >> 172.21.15.67:0/1980541576 conn(0x7f3fd80f0a70 msgr2=0x7f3fd805fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 shutdown_connections 2024-01-11T05:03:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:33.230+0000 7f3fdd9e8700 1 -- 172.21.15.67:0/1980541576 wait complete. 2024-01-11T05:03:33.622 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2024-01-11T05:03:33.622 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T05:03:33.622 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2024-01-11T05:03:33.645 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-01-11T05:03:33.645 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-11T05:03:33.705 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2024-01-11T05:03:33.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph orch host add smithi169 2024-01-11T05:03:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:32.743833+0000 mon.smithi067 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:33.132198+0000 mon.smithi067 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-01-11T05:03:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:33.135617+0000 mon.smithi067 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:33.228221+0000 mon.smithi067 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 -- 172.21.15.67:0/584105922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 msgr2=0x7f5cdc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 --2- 172.21.15.67:0/584105922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc0f69b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5cd0004760 tx=0x7f5cd0009ad0).stop 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 -- 172.21.15.67:0/584105922 shutdown_connections 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 --2- 172.21.15.67:0/584105922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc0f69b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 -- 172.21.15.67:0/584105922 >> 172.21.15.67:0/584105922 conn(0x7f5cdc0f1cd0 msgr2=0x7f5cdc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 -- 172.21.15.67:0/584105922 shutdown_connections 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 -- 172.21.15.67:0/584105922 wait complete. 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.767+0000 7f5ce2b77700 1 Processor -- start 2024-01-11T05:03:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce2b77700 1 -- start start 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce2b77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce2b77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc069f00 con 0x7f5cdc0f6590 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce0913700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce0913700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45186/0 (socket says 172.21.15.67:45186) 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce0913700 1 -- 172.21.15.67:0/3984022440 learned_addr learned my addr 172.21.15.67:0/3984022440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce0913700 1 -- 172.21.15.67:0/3984022440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd0009d70 con 0x7f5cdc0f6590 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5ce0913700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc06ba80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5cd0004170 tx=0x7f5cd0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.768+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd000e040 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.769+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc06a0a0 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.769+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5cd0008ef0 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.769+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd0014c90 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.769+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc06a530 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.769+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5cd00283f0 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.769+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc0591e0 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.770+0000 7f5cd97fa700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ccc031090 0x7f5ccc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.770+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5cd0042270 con 0x7f5cdc0f6590 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.770+0000 7f5cdbfff700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ccc031090 0x7f5ccc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.770+0000 7f5cdbfff700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ccc031090 0x7f5ccc033550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f5cc80096a0 tx=0x7f5cc8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:34.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.774+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5cd00273a0 con 0x7f5cdc0f6590 2024-01-11T05:03:34.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:34.939+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cdc0fc200 con 0x7f5ccc031090 2024-01-11T05:03:34.973 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:33.142525+0000 mgr.smithi067.srbzpc (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi067 on smithi067 2024-01-11T05:03:34.973 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:33.225582+0000 mgr.smithi067.srbzpc (mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:35.553 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi169' with addr '172.21.15.169' 2024-01-11T05:03:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.552+0000 7f5cd97fa700 1 -- 172.21.15.67:0/3984022440 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5cdc0fc200 con 0x7f5ccc031090 2024-01-11T05:03:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ccc031090 msgr2=0x7f5ccc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ccc031090 0x7f5ccc033550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f5cc80096a0 tx=0x7f5cc8006b40).stop 2024-01-11T05:03:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 msgr2=0x7f5cdc06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc06ba80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5cd0004170 tx=0x7f5cd0009ad0).stop 2024-01-11T05:03:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 shutdown_connections 2024-01-11T05:03:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ccc031090 0x7f5ccc033550 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 --2- 172.21.15.67:0/3984022440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc0f6590 0x7f5cdc06ba80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 >> 172.21.15.67:0/3984022440 conn(0x7f5cdc0f1cd0 msgr2=0x7f5cdc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 shutdown_connections 2024-01-11T05:03:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:35.554+0000 7f5ce2b77700 1 -- 172.21.15.67:0/3984022440 wait complete. 2024-01-11T05:03:35.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph orch host ls --format=json 2024-01-11T05:03:36.908 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:34.940818+0000 mgr.smithi067.srbzpc (mgr.14156) 7 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:36.908 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:35.552302+0000 mon.smithi067 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:36.908 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:35.552670+0000 mgr.smithi067.srbzpc (mgr.14156) 8 : cephadm [INF] Added host smithi169 2024-01-11T05:03:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 -- 172.21.15.67:0/1447910995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 msgr2=0x7f7c6c0f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 --2- 172.21.15.67:0/1447910995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c0f4d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7c5c009fb0 tx=0x7f7c5c009b30).stop 2024-01-11T05:03:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 -- 172.21.15.67:0/1447910995 shutdown_connections 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 --2- 172.21.15.67:0/1447910995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c0f4d10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 -- 172.21.15.67:0/1447910995 >> 172.21.15.67:0/1447910995 conn(0x7f7c6c0f0390 msgr2=0x7f7c6c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 -- 172.21.15.67:0/1447910995 shutdown_connections 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.081+0000 7f7c71890700 1 -- 172.21.15.67:0/1447910995 wait complete. 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c71890700 1 Processor -- start 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c71890700 1 -- start start 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c71890700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c1069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c71890700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c106ec0 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c1069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c6affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c1069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45208/0 (socket says 172.21.15.67:45208) 2024-01-11T05:03:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c6affd700 1 -- 172.21.15.67:0/1331204026 learned_addr learned my addr 172.21.15.67:0/1331204026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.082+0000 7f7c6affd700 1 -- 172.21.15.67:0/1331204026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c005040 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.083+0000 7f7c6affd700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c1069d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7c5c000c00 tx=0x7f7c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.083+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c5c0095f0 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.083+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7c5c007e50 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.083+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6c1070c0 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.083+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c5c014e70 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.083+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c107550 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.084+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7c5c025070 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.084+0000 7f7c637fe700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7c58031090 0x7f7c58033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.084+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7c5c041d60 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.084+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6c045bb0 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.084+0000 7f7c6a7fc700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7c58031090 0x7f7c58033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.084+0000 7f7c6a7fc700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7c58031090 0x7f7c58033550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7c540096a0 tx=0x7f7c54006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.088+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7c5c00f4e0 con 0x7f7c6c0f48f0 2024-01-11T05:03:37.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.252+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c6c0f1570 con 0x7f7c58031090 2024-01-11T05:03:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.254+0000 7f7c637fe700 1 -- 172.21.15.67:0/1331204026 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f7c6c0f1570 con 0x7f7c58031090 2024-01-11T05:03:37.254 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-01-11T05:03:37.254 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2024-01-11T05:03:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7c58031090 msgr2=0x7f7c58033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7c58031090 0x7f7c58033550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7c540096a0 tx=0x7f7c54006b40).stop 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 msgr2=0x7f7c6c1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c1069d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7c5c000c00 tx=0x7f7c5c00b040).stop 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 shutdown_connections 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7c58031090 0x7f7c58033550 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 --2- 172.21.15.67:0/1331204026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c6c0f48f0 0x7f7c6c1069d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 >> 172.21.15.67:0/1331204026 conn(0x7f7c6c0f0390 msgr2=0x7f7c6c0f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 shutdown_connections 2024-01-11T05:03:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:37.255+0000 7f7c71890700 1 -- 172.21.15.67:0/1331204026 wait complete. 2024-01-11T05:03:37.629 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-11T05:03:37.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph osd crush tunables default 2024-01-11T05:03:38.239 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:37.254234+0000 mgr.smithi067.srbzpc (mgr.14156) 9 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-11T05:03:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.740+0000 7f95219db700 1 -- 172.21.15.67:0/3865968312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 msgr2=0x7f951c0f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.740+0000 7f95219db700 1 --2- 172.21.15.67:0/3865968312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c0f67f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9510009fb0 tx=0x7f9510009b30).stop 2024-01-11T05:03:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.740+0000 7f95219db700 1 -- 172.21.15.67:0/3865968312 shutdown_connections 2024-01-11T05:03:38.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.740+0000 7f95219db700 1 --2- 172.21.15.67:0/3865968312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c0f67f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.740+0000 7f95219db700 1 -- 172.21.15.67:0/3865968312 >> 172.21.15.67:0/3865968312 conn(0x7f951c0f1b10 msgr2=0x7f951c0f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.741+0000 7f95219db700 1 -- 172.21.15.67:0/3865968312 shutdown_connections 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.741+0000 7f95219db700 1 -- 172.21.15.67:0/3865968312 wait complete. 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.741+0000 7f95219db700 1 Processor -- start 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.741+0000 7f95219db700 1 -- start start 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95219db700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95219db700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c106b30 con 0x7f951c0f63d0 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95209d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95209d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45228/0 (socket says 172.21.15.67:45228) 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95209d9700 1 -- 172.21.15.67:0/1969854276 learned_addr learned my addr 172.21.15.67:0/1969854276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95209d9700 1 -- 172.21.15.67:0/1969854276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510005040 con 0x7f951c0f63d0 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95209d9700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c106640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f95100089f0 tx=0x7f951000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9510009320 con 0x7f951c0f63d0 2024-01-11T05:03:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951c106d30 con 0x7f951c0f63d0 2024-01-11T05:03:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9510013ec0 con 0x7f951c0f63d0 2024-01-11T05:03:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9510014d30 con 0x7f951c0f63d0 2024-01-11T05:03:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.742+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c1071c0 con 0x7f951c0f63d0 2024-01-11T05:03:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.743+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f95100263a0 con 0x7f951c0f63d0 2024-01-11T05:03:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.743+0000 7f95197fa700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f950c031090 0x7f950c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.743+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c045bb0 con 0x7f951c0f63d0 2024-01-11T05:03:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.743+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9510042460 con 0x7f951c0f63d0 2024-01-11T05:03:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.743+0000 7f951bfff700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f950c031090 0x7f950c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.744+0000 7f951bfff700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f950c031090 0x7f950c033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f95080096a0 tx=0x7f9508006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:38.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.748+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f95100126f0 con 0x7f951c0f63d0 2024-01-11T05:03:38.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.913+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f951c108070 con 0x7f951c0f63d0 2024-01-11T05:03:38.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.948+0000 7f95197fa700 1 -- 172.21.15.67:0/1969854276 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9510012330 con 0x7f951c0f63d0 2024-01-11T05:03:38.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f950c031090 msgr2=0x7f950c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f950c031090 0x7f950c033550 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f95080096a0 tx=0x7f9508006b40).stop 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 msgr2=0x7f951c106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c106640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f95100089f0 tx=0x7f951000b040).stop 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 shutdown_connections 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f950c031090 0x7f950c033550 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 --2- 172.21.15.67:0/1969854276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f951c0f63d0 0x7f951c106640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 >> 172.21.15.67:0/1969854276 conn(0x7f951c0f1b10 msgr2=0x7f951c0f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 shutdown_connections 2024-01-11T05:03:38.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-01-11T05:03:38.950+0000 7f95219db700 1 -- 172.21.15.67:0/1969854276 wait complete. 2024-01-11T05:03:38.952 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-01-11T05:03:39.231 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:38.914631+0000 mon.smithi067 (mon.0) 102 : audit [INF] from='client.? 172.21.15.67:0/1969854276' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-11T05:03:39.405 INFO:tasks.cephadm:Adding mon.smithi067 on smithi067 2024-01-11T05:03:39.405 INFO:tasks.cephadm:Adding mon.smithi169 on smithi169 2024-01-11T05:03:39.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph orch apply mon '2;smithi067:172.21.15.67=smithi067;smithi169:172.21.15.169=smithi169' 2024-01-11T05:03:40.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:38.949414+0000 mon.smithi067 (mon.0) 103 : audit [INF] from='client.? 172.21.15.67:0/1969854276' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-11T05:03:40.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:38.949459+0000 mon.smithi067 (mon.0) 104 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-11T05:03:40.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.572+0000 7f88930ca700 1 -- 172.21.15.169:0/179770653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 msgr2=0x7f888c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:40.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.572+0000 7f88930ca700 1 --2- 172.21.15.169:0/179770653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c0f58a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8880009fb0 tx=0x7f8880009b30).stop 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.572+0000 7f88930ca700 1 -- 172.21.15.169:0/179770653 shutdown_connections 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.572+0000 7f88930ca700 1 --2- 172.21.15.169:0/179770653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c0f58a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.572+0000 7f88930ca700 1 -- 172.21.15.169:0/179770653 >> 172.21.15.169:0/179770653 conn(0x7f888c0f0be0 msgr2=0x7f888c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.573+0000 7f88930ca700 1 -- 172.21.15.169:0/179770653 shutdown_connections 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.573+0000 7f88930ca700 1 -- 172.21.15.169:0/179770653 wait complete. 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.573+0000 7f88930ca700 1 Processor -- start 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.573+0000 7f88930ca700 1 -- start start 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.573+0000 7f88930ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c10b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.573+0000 7f88930ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c107300 con 0x7f888c0f5480 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f8890e66700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c10b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f8890e66700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c10b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48814/0 (socket says 172.21.15.169:48814) 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f8890e66700 1 -- 172.21.15.169:0/2965306885 learned_addr learned my addr 172.21.15.169:0/2965306885 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f8890e66700 1 -- 172.21.15.169:0/2965306885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8880005040 con 0x7f888c0f5480 2024-01-11T05:03:40.575 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f8890e66700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c10b910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f8880000c00 tx=0x7f888000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88800095f0 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8880007e50 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c060300 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8880014e70 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.574+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c10be00 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.575+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8880025070 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.575+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c045bb0 con 0x7f888c0f5480 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.575+0000 7f88897fa700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f887c031090 0x7f887c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:40.577 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.575+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8880041e20 con 0x7f888c0f5480 2024-01-11T05:03:40.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.575+0000 7f888bfff700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f887c031090 0x7f887c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:40.578 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.576+0000 7f888bfff700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f887c031090 0x7f887c033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f88780096a0 tx=0x7f8878006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:40.580 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.580+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f888000f4e0 con 0x7f888c0f5480 2024-01-11T05:03:40.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.741+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 --> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi067:172.21.15.67=smithi067;smithi169:172.21.15.169=smithi169", "target": ["mon-mgr", ""]}) v1 -- 0x7f888c0f1e90 con 0x7f887c031090 2024-01-11T05:03:40.748 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.747+0000 7f88897fa700 1 -- 172.21.15.169:0/2965306885 <== mgr.14156 v2:172.21.15.67:6800/1409268088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f888c0f1e90 con 0x7f887c031090 2024-01-11T05:03:40.748 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2024-01-11T05:03:40.749 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f887c031090 msgr2=0x7f887c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f887c031090 0x7f887c033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f88780096a0 tx=0x7f8878006b40).stop 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 msgr2=0x7f888c10b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c10b910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f8880000c00 tx=0x7f888000b040).stop 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 shutdown_connections 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f887c031090 0x7f887c033550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 --2- 172.21.15.169:0/2965306885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888c0f5480 0x7f888c10b910 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 >> 172.21.15.169:0/2965306885 conn(0x7f888c0f0be0 msgr2=0x7f888c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 shutdown_connections 2024-01-11T05:03:40.750 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:40.749+0000 7f88930ca700 1 -- 172.21.15.169:0/2965306885 wait complete. 2024-01-11T05:03:41.120 DEBUG:teuthology.orchestra.run.smithi169:mon.smithi169> sudo journalctl -f -n 0 -u ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169.service 2024-01-11T05:03:41.122 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:41.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:41.152 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:-- Logs begin at Thu 2024-01-11 04:51:40 UTC. -- 2024-01-11T05:03:42.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:40.744137+0000 mgr.smithi067.srbzpc (mgr.14156) 10 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi067:172.21.15.67=smithi067;smithi169:172.21.15.169=smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-01-11T05:03:42.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:03:40.745631+0000 mgr.smithi067.srbzpc (mgr.14156) 11 : cephadm [INF] Saving service mon spec with placement smithi067:172.21.15.67=smithi067;smithi169:172.21.15.169=smithi169;count:2 2024-01-11T05:03:42.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:40.748325+0000 mon.smithi067 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:03:42.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 -- 172.21.15.169:0/1336503130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 msgr2=0x7f0a280f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:42.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 --2- 172.21.15.169:0/1336503130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a280f58d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0a18009fb0 tx=0x7f0a18009b30).stop 2024-01-11T05:03:42.263 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 -- 172.21.15.169:0/1336503130 shutdown_connections 2024-01-11T05:03:42.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 --2- 172.21.15.169:0/1336503130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a280f58d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:42.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 -- 172.21.15.169:0/1336503130 >> 172.21.15.169:0/1336503130 conn(0x7f0a280f0c10 msgr2=0x7f0a280f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:42.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 -- 172.21.15.169:0/1336503130 shutdown_connections 2024-01-11T05:03:42.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.262+0000 7f0a2dabe700 1 -- 172.21.15.169:0/1336503130 wait complete. 2024-01-11T05:03:42.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a2dabe700 1 Processor -- start 2024-01-11T05:03:42.264 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a2dabe700 1 -- start start 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a2dabe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a28109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a2dabe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a28060300 con 0x7f0a280f54b0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a277fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a28109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a277fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a28109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48838/0 (socket says 172.21.15.169:48838) 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.263+0000 7f0a277fe700 1 -- 172.21.15.169:0/3440842532 learned_addr learned my addr 172.21.15.169:0/3440842532 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a277fe700 1 -- 172.21.15.169:0/3440842532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18005040 con 0x7f0a280f54b0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a277fe700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a28109510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0a180089f0 tx=0x7f0a1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a18009320 con 0x7f0a280f54b0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a28060500 con 0x7f0a280f54b0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0a18013ec0 con 0x7f0a280f54b0 2024-01-11T05:03:42.266 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a18014d30 con 0x7f0a280f54b0 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.264+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a28109bc0 con 0x7f0a280f54b0 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.265+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0a18014420 con 0x7f0a280f54b0 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.265+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a280591e0 con 0x7f0a280f54b0 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.265+0000 7f0a0ffff700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0a100310e0 0x7f0a100335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.265+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0a180424c0 con 0x7f0a280f54b0 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.265+0000 7f0a26ffd700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0a100310e0 0x7f0a100335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:42.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.266+0000 7f0a26ffd700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0a100310e0 0x7f0a100335a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f0a1c0096a0 tx=0x7f0a1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:42.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.270+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0a1801d980 con 0x7f0a280f54b0 2024-01-11T05:03:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.475+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a28045bb0 con 0x7f0a280f54b0 2024-01-11T05:03:42.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.476+0000 7f0a0ffff700 1 -- 172.21.15.169:0/3440842532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a180242b0 con 0x7f0a280f54b0 2024-01-11T05:03:42.476 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:42.476 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0a100310e0 msgr2=0x7f0a100335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0a100310e0 0x7f0a100335a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f0a1c0096a0 tx=0x7f0a1c006b40).stop 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 msgr2=0x7f0a28109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a28109510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0a180089f0 tx=0x7f0a1800b040).stop 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 shutdown_connections 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0a100310e0 0x7f0a100335a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 --2- 172.21.15.169:0/3440842532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a280f54b0 0x7f0a28109510 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 >> 172.21.15.169:0/3440842532 conn(0x7f0a280f0c10 msgr2=0x7f0a280f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:42.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 shutdown_connections 2024-01-11T05:03:42.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:42.477+0000 7f0a2dabe700 1 -- 172.21.15.169:0/3440842532 wait complete. 2024-01-11T05:03:42.479 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:42.269904+0000 mgr.smithi067.srbzpc (mgr.14156) 12 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:42.477124+0000 mon.smithi067 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.169:0/3440842532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:03:43.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:43.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:45.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 -- 172.21.15.169:0/2745692075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 msgr2=0x7f6f300f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 --2- 172.21.15.169:0/2745692075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f300f9050 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6f20009fb0 tx=0x7f6f20009b30).stop 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 -- 172.21.15.169:0/2745692075 shutdown_connections 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 --2- 172.21.15.169:0/2745692075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f300f9050 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 -- 172.21.15.169:0/2745692075 >> 172.21.15.169:0/2745692075 conn(0x7f6f300f0a30 msgr2=0x7f6f300f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 -- 172.21.15.169:0/2745692075 shutdown_connections 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 -- 172.21.15.169:0/2745692075 wait complete. 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.012+0000 7f6f34f4b700 1 Processor -- start 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f34f4b700 1 -- start start 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f34f4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f301065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f34f4b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f30106aa0 con 0x7f6f300602c0 2024-01-11T05:03:45.014 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f2f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f301065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:45.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f2f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f301065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48852/0 (socket says 172.21.15.169:48852) 2024-01-11T05:03:45.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f2f7fe700 1 -- 172.21.15.169:0/1384363534 learned_addr learned my addr 172.21.15.169:0/1384363534 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:45.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f2f7fe700 1 -- 172.21.15.169:0/1384363534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f20005040 con 0x7f6f300602c0 2024-01-11T05:03:45.015 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f2f7fe700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f301065b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6f20013fd0 tx=0x7f6f20009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f20018070 con 0x7f6f300602c0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.013+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6f20007b10 con 0x7f6f300602c0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.014+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f2001c3c0 con 0x7f6f300602c0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.014+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f30106ca0 con 0x7f6f300602c0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.014+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f30107130 con 0x7f6f300602c0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.014+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6f20007c80 con 0x7f6f300602c0 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.014+0000 7f6f17fff700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f180310e0 0x7f6f180335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:45.016 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.014+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6f200429c0 con 0x7f6f300602c0 2024-01-11T05:03:45.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.015+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f30045bb0 con 0x7f6f300602c0 2024-01-11T05:03:45.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.015+0000 7f6f2effd700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f180310e0 0x7f6f180335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:45.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.015+0000 7f6f2effd700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f180310e0 0x7f6f180335a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f6f240096a0 tx=0x7f6f24006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:45.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.020+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6f2001cb10 con 0x7f6f300602c0 2024-01-11T05:03:45.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.226+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f30107de0 con 0x7f6f300602c0 2024-01-11T05:03:45.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.227+0000 7f6f17fff700 1 -- 172.21.15.169:0/1384363534 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f20025b40 con 0x7f6f300602c0 2024-01-11T05:03:45.227 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:45.228 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f180310e0 msgr2=0x7f6f180335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f180310e0 0x7f6f180335a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f6f240096a0 tx=0x7f6f24006b40).stop 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 msgr2=0x7f6f301065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f301065b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6f20013fd0 tx=0x7f6f20009b30).stop 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 shutdown_connections 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f180310e0 0x7f6f180335a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 --2- 172.21.15.169:0/1384363534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f300602c0 0x7f6f301065b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 >> 172.21.15.169:0/1384363534 conn(0x7f6f300f0a30 msgr2=0x7f6f300f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 shutdown_connections 2024-01-11T05:03:45.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:45.228+0000 7f6f34f4b700 1 -- 172.21.15.169:0/1384363534 wait complete. 2024-01-11T05:03:45.230 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:45.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:44.270193+0000 mgr.smithi067.srbzpc (mgr.14156) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:46.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:45.227892+0000 mon.smithi067 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.169:0/1384363534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:03:46.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:46.605 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:47.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:46.270490+0000 mgr.smithi067.srbzpc (mgr.14156) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 -- 172.21.15.169:0/585029565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 msgr2=0x7f96980f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 --2- 172.21.15.169:0/585029565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96980f69b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f968c009fb0 tx=0x7f968c009b30).stop 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 -- 172.21.15.169:0/585029565 shutdown_connections 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 --2- 172.21.15.169:0/585029565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96980f69b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 -- 172.21.15.169:0/585029565 >> 172.21.15.169:0/585029565 conn(0x7f96980f1cf0 msgr2=0x7f96980f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 -- 172.21.15.169:0/585029565 shutdown_connections 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.752+0000 7f969d72c700 1 -- 172.21.15.169:0/585029565 wait complete. 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f969d72c700 1 Processor -- start 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f969d72c700 1 -- start start 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f969d72c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96981067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:47.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f969d72c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9698106cc0 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f9696ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96981067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f9696ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96981067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48866/0 (socket says 172.21.15.169:48866) 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.753+0000 7f9696ffd700 1 -- 172.21.15.169:0/905836961 learned_addr learned my addr 172.21.15.169:0/905836961 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f9696ffd700 1 -- 172.21.15.169:0/905836961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c005040 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f9696ffd700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96981067d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f968c0089f0 tx=0x7f968c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968c009320 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9698106ec0 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f968c013ec0 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968c014d30 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.754+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9698107350 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.755+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f968c014420 con 0x7f96980f6590 2024-01-11T05:03:47.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.755+0000 7f967f7fe700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9680031090 0x7f9680033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:47.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.755+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9698045bb0 con 0x7f96980f6590 2024-01-11T05:03:47.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.755+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f968c0424a0 con 0x7f96980f6590 2024-01-11T05:03:47.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.755+0000 7f96967fc700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9680031090 0x7f9680033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:47.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.756+0000 7f96967fc700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9680031090 0x7f9680033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f96880096a0 tx=0x7f9688006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:47.760 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.760+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f968c014750 con 0x7f96980f6590 2024-01-11T05:03:47.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.969+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9698108150 con 0x7f96980f6590 2024-01-11T05:03:47.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.970+0000 7f967f7fe700 1 -- 172.21.15.169:0/905836961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f968c01d7f0 con 0x7f96980f6590 2024-01-11T05:03:47.971 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:47.971 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:47.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9680031090 msgr2=0x7f9680033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:47.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9680031090 0x7f9680033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f96880096a0 tx=0x7f9688006b40).stop 2024-01-11T05:03:47.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 msgr2=0x7f96981067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96981067d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f968c0089f0 tx=0x7f968c00b040).stop 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 shutdown_connections 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9680031090 0x7f9680033550 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 --2- 172.21.15.169:0/905836961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96980f6590 0x7f96981067d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 >> 172.21.15.169:0/905836961 conn(0x7f96980f1cf0 msgr2=0x7f96980f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 shutdown_connections 2024-01-11T05:03:47.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:47.971+0000 7f969d72c700 1 -- 172.21.15.169:0/905836961 wait complete. 2024-01-11T05:03:47.974 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:48.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:48 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:47.971077+0000 mon.smithi067 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.169:0/905836961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:03:49.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:48.270732+0000 mgr.smithi067.srbzpc (mgr.14156) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:49.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:49.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.580+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/4173251985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 msgr2=0x7fd3e40f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.580+0000 7fd3e9a8c700 1 --2- 172.21.15.169:0/4173251985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e40f6800 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fd3d8009fb0 tx=0x7fd3d8009b30).stop 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/4173251985 shutdown_connections 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 --2- 172.21.15.169:0/4173251985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e40f6800 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/4173251985 >> 172.21.15.169:0/4173251985 conn(0x7fd3e40f1b40 msgr2=0x7fd3e40f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/4173251985 shutdown_connections 2024-01-11T05:03:50.582 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/4173251985 wait complete. 2024-01-11T05:03:50.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 Processor -- start 2024-01-11T05:03:50.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 -- start start 2024-01-11T05:03:50.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e4106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:50.583 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.581+0000 7fd3e9a8c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4106b10 con 0x7fd3e40f63e0 2024-01-11T05:03:50.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e8a8a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e4106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:50.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e8a8a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e4106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54460/0 (socket says 172.21.15.169:54460) 2024-01-11T05:03:50.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e8a8a700 1 -- 172.21.15.169:0/2604601007 learned_addr learned my addr 172.21.15.169:0/2604601007 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:50.584 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e8a8a700 1 -- 172.21.15.169:0/2604601007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8005040 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e8a8a700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e4106620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd3d80089f0 tx=0x7fd3d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3d8009320 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.582+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e4106d10 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd3d8013ec0 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3d8014d30 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e41071a0 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd3d8014420 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e17fa700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3d4031090 0x7fd3d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd3d80424a0 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.583+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e4045bb0 con 0x7fd3e40f63e0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.584+0000 7fd3e3fff700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3d4031090 0x7fd3d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:50.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.584+0000 7fd3e3fff700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3d4031090 0x7fd3d4033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd3d00096a0 tx=0x7fd3d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:50.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.588+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd3d8014750 con 0x7fd3e40f63e0 2024-01-11T05:03:50.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.798+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3e40fbfc0 con 0x7fd3e40f63e0 2024-01-11T05:03:50.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.799+0000 7fd3e17fa700 1 -- 172.21.15.169:0/2604601007 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3d801d7f0 con 0x7fd3e40f63e0 2024-01-11T05:03:50.800 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:50.800 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3d4031090 msgr2=0x7fd3d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3d4031090 0x7fd3d4033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd3d00096a0 tx=0x7fd3d0006b40).stop 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 msgr2=0x7fd3e4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e4106620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd3d80089f0 tx=0x7fd3d800b040).stop 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 shutdown_connections 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3d4031090 0x7fd3d4033550 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:50.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 --2- 172.21.15.169:0/2604601007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e40f63e0 0x7fd3e4106620 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:50.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.800+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 >> 172.21.15.169:0/2604601007 conn(0x7fd3e40f1b40 msgr2=0x7fd3e40f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:50.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.801+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 shutdown_connections 2024-01-11T05:03:50.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:50.801+0000 7fd3e9a8c700 1 -- 172.21.15.169:0/2604601007 wait complete. 2024-01-11T05:03:50.802 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:50.270988+0000 mgr.smithi067.srbzpc (mgr.14156) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:50.800222+0000 mon.smithi067 (mon.0) 109 : audit [DBG] from='client.? 172.21.15.169:0/2604601007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:03:52.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:52.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:53.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 -- 172.21.15.169:0/3793621415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 msgr2=0x7f9efc0f5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:53.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 --2- 172.21.15.169:0/3793621415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0f5cc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9ef0009fb0 tx=0x7f9ef0009b30).stop 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 -- 172.21.15.169:0/3793621415 shutdown_connections 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 --2- 172.21.15.169:0/3793621415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0f5cc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 -- 172.21.15.169:0/3793621415 >> 172.21.15.169:0/3793621415 conn(0x7f9efc0f0190 msgr2=0x7f9efc0f25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 -- 172.21.15.169:0/3793621415 shutdown_connections 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.387+0000 7f9f01804700 1 -- 172.21.15.169:0/3793621415 wait complete. 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f01804700 1 Processor -- start 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f01804700 1 -- start start 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f01804700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f01804700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0fb730 con 0x7f9efc0f58a0 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f00802700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f00802700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54468/0 (socket says 172.21.15.169:54468) 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f00802700 1 -- 172.21.15.169:0/3317071045 learned_addr learned my addr 172.21.15.169:0/3317071045 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.388+0000 7f9f00802700 1 -- 172.21.15.169:0/3317071045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef0005040 con 0x7f9efc0f58a0 2024-01-11T05:03:53.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.389+0000 7f9f00802700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0fd2b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f9ef0000c00 tx=0x7f9ef000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:53.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.389+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef00095f0 con 0x7f9efc0f58a0 2024-01-11T05:03:53.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.389+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9ef0007e50 con 0x7f9efc0f58a0 2024-01-11T05:03:53.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.389+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0fb8d0 con 0x7f9efc0f58a0 2024-01-11T05:03:53.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.389+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef0014e70 con 0x7f9efc0f58a0 2024-01-11T05:03:53.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.389+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc0fbd60 con 0x7f9efc0f58a0 2024-01-11T05:03:53.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.390+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9ef0025070 con 0x7f9efc0f58a0 2024-01-11T05:03:53.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.390+0000 7f9ef97fa700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eec031090 0x7f9eec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:53.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.390+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc0591e0 con 0x7f9efc0f58a0 2024-01-11T05:03:53.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.390+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9ef0041e50 con 0x7f9efc0f58a0 2024-01-11T05:03:53.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.390+0000 7f9efbfff700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eec031090 0x7f9eec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:53.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.391+0000 7f9efbfff700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eec031090 0x7f9eec033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9ee80096a0 tx=0x7f9ee8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:53.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.395+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9ef0026350 con 0x7f9efc0f58a0 2024-01-11T05:03:53.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.600+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9efc045bb0 con 0x7f9efc0f58a0 2024-01-11T05:03:53.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.601+0000 7f9ef97fa700 1 -- 172.21.15.169:0/3317071045 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ef0012330 con 0x7f9efc0f58a0 2024-01-11T05:03:53.602 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:53.602 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:53.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eec031090 msgr2=0x7f9eec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:53.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eec031090 0x7f9eec033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9ee80096a0 tx=0x7f9ee8006b40).stop 2024-01-11T05:03:53.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 msgr2=0x7f9efc0fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:53.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0fd2b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f9ef0000c00 tx=0x7f9ef000b040).stop 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 shutdown_connections 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eec031090 0x7f9eec033550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 --2- 172.21.15.169:0/3317071045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9efc0f58a0 0x7f9efc0fd2b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 >> 172.21.15.169:0/3317071045 conn(0x7f9efc0f0190 msgr2=0x7f9efc0f4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 shutdown_connections 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:53.603+0000 7f9f01804700 1 -- 172.21.15.169:0/3317071045 wait complete. 2024-01-11T05:03:53.604 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:53.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:52.271310+0000 mgr.smithi067.srbzpc (mgr.14156) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:54.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:54 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:53.602288+0000 mon.smithi067 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.169:0/3317071045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:03:54.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:54.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:55.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:54.271731+0000 mgr.smithi067.srbzpc (mgr.14156) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:56.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 -- 172.21.15.169:0/292218642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 msgr2=0x7f880c0f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:56.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 --2- 172.21.15.169:0/292218642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c0f5030 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f8808009fb0 tx=0x7f8808009b30).stop 2024-01-11T05:03:56.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 -- 172.21.15.169:0/292218642 shutdown_connections 2024-01-11T05:03:56.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 --2- 172.21.15.169:0/292218642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c0f5030 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:56.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 -- 172.21.15.169:0/292218642 >> 172.21.15.169:0/292218642 conn(0x7f880c0f0390 msgr2=0x7f880c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 -- 172.21.15.169:0/292218642 shutdown_connections 2024-01-11T05:03:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 -- 172.21.15.169:0/292218642 wait complete. 2024-01-11T05:03:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 Processor -- start 2024-01-11T05:03:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.212+0000 7f88146fb700 1 -- start start 2024-01-11T05:03:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f88146fb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c105700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:56.214 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f88146fb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c105bf0 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f8812497700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f8812497700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54478/0 (socket says 172.21.15.169:54478) 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f8812497700 1 -- 172.21.15.169:0/3497379906 learned_addr learned my addr 172.21.15.169:0/3497379906 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f8812497700 1 -- 172.21.15.169:0/3497379906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8808005040 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.213+0000 7f8812497700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c105700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f88080089f0 tx=0x7f880800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.214+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8808009320 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.214+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c105df0 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.214+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8808013ec0 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.214+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8808014d30 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.214+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c106280 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.215+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f88080095f0 con 0x7f880c0f4c10 2024-01-11T05:03:56.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.215+0000 7f8802ffd700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f87f80310e0 0x7f87f80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:56.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.215+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f88080423a0 con 0x7f880c0f4c10 2024-01-11T05:03:56.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.215+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c045bb0 con 0x7f880c0f4c10 2024-01-11T05:03:56.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.215+0000 7f8811c96700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f87f80310e0 0x7f87f80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:56.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.215+0000 7f8811c96700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f87f80310e0 0x7f87f80335a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f87fc0096a0 tx=0x7f87fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:56.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.220+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f880801d7f0 con 0x7f880c0f4c10 2024-01-11T05:03:56.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.425+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f880c0fa850 con 0x7f880c0f4c10 2024-01-11T05:03:56.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.425+0000 7f8802ffd700 1 -- 172.21.15.169:0/3497379906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f880801d7f0 con 0x7f880c0f4c10 2024-01-11T05:03:56.426 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:56.426 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f87f80310e0 msgr2=0x7f87f80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f87f80310e0 0x7f87f80335a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f87fc0096a0 tx=0x7f87fc006b40).stop 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 msgr2=0x7f880c105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c105700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f88080089f0 tx=0x7f880800b040).stop 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 shutdown_connections 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f87f80310e0 0x7f87f80335a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 --2- 172.21.15.169:0/3497379906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f880c0f4c10 0x7f880c105700 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:56.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 >> 172.21.15.169:0/3497379906 conn(0x7f880c0f0390 msgr2=0x7f880c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:56.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 shutdown_connections 2024-01-11T05:03:56.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:56.427+0000 7f88146fb700 1 -- 172.21.15.169:0/3497379906 wait complete. 2024-01-11T05:03:56.429 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:56.272072+0000 mgr.smithi067.srbzpc (mgr.14156) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:56.426797+0000 mon.smithi067 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.169:0/3497379906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:03:57.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:03:57.800 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:03:58.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 -- 172.21.15.169:0/273680144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 msgr2=0x7ff5bc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:58.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 --2- 172.21.15.169:0/273680144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc0f58a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff5ac009fb0 tx=0x7ff5ac009b30).stop 2024-01-11T05:03:58.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 -- 172.21.15.169:0/273680144 shutdown_connections 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 --2- 172.21.15.169:0/273680144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc0f58a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 -- 172.21.15.169:0/273680144 >> 172.21.15.169:0/273680144 conn(0x7ff5bc0f0be0 msgr2=0x7ff5bc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 -- 172.21.15.169:0/273680144 shutdown_connections 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.949+0000 7ff5c28a5700 1 -- 172.21.15.169:0/273680144 wait complete. 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.950+0000 7ff5c28a5700 1 Processor -- start 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.950+0000 7ff5c28a5700 1 -- start start 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.950+0000 7ff5c28a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.950+0000 7ff5c28a5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc105be0 con 0x7ff5bc0f5480 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54482/0 (socket says 172.21.15.169:54482) 2024-01-11T05:03:58.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5bbfff700 1 -- 172.21.15.169:0/1266174559 learned_addr learned my addr 172.21.15.169:0/1266174559 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5bbfff700 1 -- 172.21.15.169:0/1266174559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac005040 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5bbfff700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc1056f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff5ac0089f0 tx=0x7ff5ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5ac009320 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5bc105de0 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff5ac013ec0 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5ac014d30 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.951+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5bc106270 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.952+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff5ac0263a0 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.952+0000 7ff5b8ff9700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5a4031090 0x7ff5a4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.952+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5bc045bb0 con 0x7ff5bc0f5480 2024-01-11T05:03:58.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.952+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff5ac042410 con 0x7ff5bc0f5480 2024-01-11T05:03:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.953+0000 7ff5bb7fe700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5a4031090 0x7ff5a4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:03:58.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.953+0000 7ff5bb7fe700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5a4031090 0x7ff5a4033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff5b00096a0 tx=0x7ff5b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:03:58.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:58.957+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff5ac012510 con 0x7ff5bc0f5480 2024-01-11T05:03:59.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.164+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5bc0fb060 con 0x7ff5bc0f5480 2024-01-11T05:03:59.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.165+0000 7ff5b8ff9700 1 -- 172.21.15.169:0/1266174559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5ac012330 con 0x7ff5bc0f5480 2024-01-11T05:03:59.165 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:03:59.166 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:03:59.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5a4031090 msgr2=0x7ff5a4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:59.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5a4031090 0x7ff5a4033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff5b00096a0 tx=0x7ff5b0006b40).stop 2024-01-11T05:03:59.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 msgr2=0x7ff5bc1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:03:59.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc1056f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff5ac0089f0 tx=0x7ff5ac00b040).stop 2024-01-11T05:03:59.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 shutdown_connections 2024-01-11T05:03:59.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5a4031090 0x7ff5a4033550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:59.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 --2- 172.21.15.169:0/1266174559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc0f5480 0x7ff5bc1056f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:03:59.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 >> 172.21.15.169:0/1266174559 conn(0x7ff5bc0f0be0 msgr2=0x7ff5bc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:03:59.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 shutdown_connections 2024-01-11T05:03:59.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:03:59.166+0000 7ff5c28a5700 1 -- 172.21.15.169:0/1266174559 wait complete. 2024-01-11T05:03:59.168 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:03:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:03:58.272410+0000 mgr.smithi067.srbzpc (mgr.14156) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:03:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:03:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:03:59.165913+0000 mon.smithi067 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.169:0/1266174559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:00.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:00.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:01.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 -- 172.21.15.169:0/4025949784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 msgr2=0x7ff0f40f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 --2- 172.21.15.169:0/4025949784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f40f58d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff0e4009fb0 tx=0x7ff0e4009b30).stop 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 -- 172.21.15.169:0/4025949784 shutdown_connections 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 --2- 172.21.15.169:0/4025949784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f40f58d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 -- 172.21.15.169:0/4025949784 >> 172.21.15.169:0/4025949784 conn(0x7ff0f40f0c10 msgr2=0x7ff0f40f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 -- 172.21.15.169:0/4025949784 shutdown_connections 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.693+0000 7ff0fb2df700 1 -- 172.21.15.169:0/4025949784 wait complete. 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0fb2df700 1 Processor -- start 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0fb2df700 1 -- start start 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0fb2df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0fb2df700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f4106ca0 con 0x7ff0f40f54b0 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0f907b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:01.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0f907b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46086/0 (socket says 172.21.15.169:46086) 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0f907b700 1 -- 172.21.15.169:0/3415914789 learned_addr learned my addr 172.21.15.169:0/3415914789 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0f907b700 1 -- 172.21.15.169:0/3415914789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4005040 con 0x7ff0f40f54b0 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.694+0000 7ff0f907b700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f41067b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff0e40089f0 tx=0x7ff0e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.695+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e4009320 con 0x7ff0f40f54b0 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.695+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f4106ea0 con 0x7ff0f40f54b0 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.695+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff0e4013ec0 con 0x7ff0f40f54b0 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.695+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e4014d30 con 0x7ff0f40f54b0 2024-01-11T05:04:01.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.695+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f4107330 con 0x7ff0f40f54b0 2024-01-11T05:04:01.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.696+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff0e4014420 con 0x7ff0f40f54b0 2024-01-11T05:04:01.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.696+0000 7ff0e9ffb700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff0e0031090 0x7ff0e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:01.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.696+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f4045bb0 con 0x7ff0f40f54b0 2024-01-11T05:04:01.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.696+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff0e40424c0 con 0x7ff0f40f54b0 2024-01-11T05:04:01.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.696+0000 7ff0f887a700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff0e0031090 0x7ff0e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:01.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.697+0000 7ff0f887a700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff0e0031090 0x7ff0e0033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ff0f00096a0 tx=0x7ff0f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:01.701 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.702+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff0e4014750 con 0x7ff0f40f54b0 2024-01-11T05:04:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:00.272815+0000 mgr.smithi067.srbzpc (mgr.14156) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:01.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.910+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0f4108130 con 0x7ff0f40f54b0 2024-01-11T05:04:01.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.911+0000 7ff0e9ffb700 1 -- 172.21.15.169:0/3415914789 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0e401d380 con 0x7ff0f40f54b0 2024-01-11T05:04:01.911 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:01.911 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:01.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff0e0031090 msgr2=0x7ff0e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:01.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff0e0031090 0x7ff0e0033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ff0f00096a0 tx=0x7ff0f0006b40).stop 2024-01-11T05:04:01.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 msgr2=0x7ff0f41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f41067b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff0e40089f0 tx=0x7ff0e400b040).stop 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 shutdown_connections 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff0e0031090 0x7ff0e0033550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 --2- 172.21.15.169:0/3415914789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f40f54b0 0x7ff0f41067b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 >> 172.21.15.169:0/3415914789 conn(0x7ff0f40f0c10 msgr2=0x7ff0f405ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 shutdown_connections 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:01.913+0000 7ff0fb2df700 1 -- 172.21.15.169:0/3415914789 wait complete. 2024-01-11T05:04:01.913 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:02.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:01.911435+0000 mon.smithi067 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.169:0/3415914789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:03.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:03.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:02.273053+0000 mgr.smithi067.srbzpc (mgr.14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 -- 172.21.15.169:0/1648398874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 msgr2=0x7f65340f4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 --2- 172.21.15.169:0/1648398874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f65340f4ab0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6528009fb0 tx=0x7f6528009b30).stop 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 -- 172.21.15.169:0/1648398874 shutdown_connections 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 --2- 172.21.15.169:0/1648398874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f65340f4ab0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 -- 172.21.15.169:0/1648398874 >> 172.21.15.169:0/1648398874 conn(0x7f65340f0190 msgr2=0x7f65340f25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 -- 172.21.15.169:0/1648398874 shutdown_connections 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.426+0000 7f6539433700 1 -- 172.21.15.169:0/1648398874 wait complete. 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6539433700 1 Processor -- start 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6539433700 1 -- start start 2024-01-11T05:04:04.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6539433700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f6534106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:04.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6539433700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6534106b30 con 0x7f65340f4690 2024-01-11T05:04:04.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6533fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f6534106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:04.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6533fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f6534106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46094/0 (socket says 172.21.15.169:46094) 2024-01-11T05:04:04.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.427+0000 7f6533fff700 1 -- 172.21.15.169:0/3733715485 learned_addr learned my addr 172.21.15.169:0/3733715485 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:04.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6533fff700 1 -- 172.21.15.169:0/3733715485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6528005040 con 0x7f65340f4690 2024-01-11T05:04:04.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6533fff700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f6534106640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f65280089f0 tx=0x7f652800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:04.429 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6528009320 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6528013ec0 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6534106d30 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6528014d30 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.428+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65341071c0 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.429+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6528014420 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.429+0000 7f6530ff9700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f651c031090 0x7f651c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.429+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6534045bb0 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.429+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6528042480 con 0x7f65340f4690 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.429+0000 7f65337fe700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f651c031090 0x7f651c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:04.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.430+0000 7f65337fe700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f651c031090 0x7f651c033550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f65240096a0 tx=0x7f6524006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:04.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.434+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f652801d380 con 0x7f65340f4690 2024-01-11T05:04:04.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.644+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65340fbfb0 con 0x7f65340f4690 2024-01-11T05:04:04.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.644+0000 7f6530ff9700 1 -- 172.21.15.169:0/3733715485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f652800f340 con 0x7f65340f4690 2024-01-11T05:04:04.644 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:04.644 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:04.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f651c031090 msgr2=0x7f651c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f651c031090 0x7f651c033550 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f65240096a0 tx=0x7f6524006b40).stop 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 msgr2=0x7f6534106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f6534106640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f65280089f0 tx=0x7f652800b040).stop 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 shutdown_connections 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f651c031090 0x7f651c033550 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 --2- 172.21.15.169:0/3733715485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65340f4690 0x7f6534106640 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 >> 172.21.15.169:0/3733715485 conn(0x7f65340f0190 msgr2=0x7f65340f0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 shutdown_connections 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:04.646+0000 7f6539433700 1 -- 172.21.15.169:0/3733715485 wait complete. 2024-01-11T05:04:04.646 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:05.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:04.273320+0000 mgr.smithi067.srbzpc (mgr.14156) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:05.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:04.644642+0000 mon.smithi067 (mon.0) 114 : audit [DBG] from='client.? 172.21.15.169:0/3733715485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:06.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:06.022 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:07.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 -- 172.21.15.169:0/1806355377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 msgr2=0x7f2dd40f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 --2- 172.21.15.169:0/1806355377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd40f5020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2dc8009fb0 tx=0x7f2dc8009b30).stop 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 -- 172.21.15.169:0/1806355377 shutdown_connections 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 --2- 172.21.15.169:0/1806355377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd40f5020 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 -- 172.21.15.169:0/1806355377 >> 172.21.15.169:0/1806355377 conn(0x7f2dd40f0360 msgr2=0x7f2dd40f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 -- 172.21.15.169:0/1806355377 shutdown_connections 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.149+0000 7f2dd8997700 1 -- 172.21.15.169:0/1806355377 wait complete. 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd8997700 1 Processor -- start 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd8997700 1 -- start start 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd8997700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd8997700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd4106cd0 con 0x7f2dd40f4c00 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46114/0 (socket says 172.21.15.169:46114) 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.150+0000 7f2dd259c700 1 -- 172.21.15.169:0/2369069140 learned_addr learned my addr 172.21.15.169:0/2369069140 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:07.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dd259c700 1 -- 172.21.15.169:0/2369069140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8005040 con 0x7f2dd40f4c00 2024-01-11T05:04:07.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dd259c700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd41067e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f2dc8000c00 tx=0x7f2dc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:07.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc80095f0 con 0x7f2dd40f4c00 2024-01-11T05:04:07.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2dc8007e50 con 0x7f2dd40f4c00 2024-01-11T05:04:07.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd4106ed0 con 0x7f2dd40f4c00 2024-01-11T05:04:07.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dc8014e70 con 0x7f2dd40f4c00 2024-01-11T05:04:07.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.151+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd4107360 con 0x7f2dd40f4c00 2024-01-11T05:04:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.152+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2dc8025070 con 0x7f2dd40f4c00 2024-01-11T05:04:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.152+0000 7f2dc2ffd700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2dbc031090 0x7f2dbc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.152+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd4045bb0 con 0x7f2dd40f4c00 2024-01-11T05:04:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.152+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2dc8041e20 con 0x7f2dd40f4c00 2024-01-11T05:04:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.152+0000 7f2dd1d9b700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2dbc031090 0x7f2dbc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:07.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.152+0000 7f2dd1d9b700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2dbc031090 0x7f2dbc033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f2dc40096a0 tx=0x7f2dc4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:07.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.157+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2dc800f4e0 con 0x7f2dd40f4c00 2024-01-11T05:04:07.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.364+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2dd4100880 con 0x7f2dd40f4c00 2024-01-11T05:04:07.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.365+0000 7f2dc2ffd700 1 -- 172.21.15.169:0/2369069140 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2dc800f300 con 0x7f2dd40f4c00 2024-01-11T05:04:07.365 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:07.365 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:07.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2dbc031090 msgr2=0x7f2dbc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:07.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2dbc031090 0x7f2dbc033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f2dc40096a0 tx=0x7f2dc4006b40).stop 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 msgr2=0x7f2dd41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd41067e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f2dc8000c00 tx=0x7f2dc800b040).stop 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 shutdown_connections 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2dbc031090 0x7f2dbc033550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 --2- 172.21.15.169:0/2369069140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd40f4c00 0x7f2dd41067e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 >> 172.21.15.169:0/2369069140 conn(0x7f2dd40f0360 msgr2=0x7f2dd40f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 shutdown_connections 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:07.366+0000 7f2dd8997700 1 -- 172.21.15.169:0/2369069140 wait complete. 2024-01-11T05:04:07.367 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:07.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:06.273578+0000 mgr.smithi067.srbzpc (mgr.14156) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:08.742 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:08.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:08.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:07.365155+0000 mon.smithi067 (mon.0) 115 : audit [DBG] from='client.? 172.21.15.169:0/2369069140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:08.273922+0000 mgr.smithi067.srbzpc (mgr.14156) 25 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 -- 172.21.15.169:0/2509710561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 msgr2=0x7f6eac0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 --2- 172.21.15.169:0/2509710561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac0f58d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6e94009fb0 tx=0x7f6e94009b30).stop 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 -- 172.21.15.169:0/2509710561 shutdown_connections 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 --2- 172.21.15.169:0/2509710561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac0f58d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 -- 172.21.15.169:0/2509710561 >> 172.21.15.169:0/2509710561 conn(0x7f6eac0f0c10 msgr2=0x7f6eac0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 -- 172.21.15.169:0/2509710561 shutdown_connections 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.994+0000 7f6eb092e700 1 -- 172.21.15.169:0/2509710561 wait complete. 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eb092e700 1 Processor -- start 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eb092e700 1 -- start start 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eb092e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac106840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eb092e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac106d30 con 0x7f6eac0f54b0 2024-01-11T05:04:09.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eaa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eaa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39122/0 (socket says 172.21.15.169:39122) 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eaa59c700 1 -- 172.21.15.169:0/3183017626 learned_addr learned my addr 172.21.15.169:0/3183017626 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.995+0000 7f6eaa59c700 1 -- 172.21.15.169:0/3183017626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e94005040 con 0x7f6eac0f54b0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.996+0000 7f6eaa59c700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac106840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f6e94009f80 tx=0x7f6e9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.996+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e94009320 con 0x7f6eac0f54b0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.996+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac106f30 con 0x7f6eac0f54b0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.996+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6e94013380 con 0x7f6eac0f54b0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.996+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e9401ebd0 con 0x7f6eac0f54b0 2024-01-11T05:04:09.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.996+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac1073c0 con 0x7f6eac0f54b0 2024-01-11T05:04:09.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.997+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6e9401e420 con 0x7f6eac0f54b0 2024-01-11T05:04:09.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.997+0000 7f6ea2ffd700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6e98031090 0x7f6e98033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:09.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.997+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6e94042220 con 0x7f6eac0f54b0 2024-01-11T05:04:09.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.997+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac045bb0 con 0x7f6eac0f54b0 2024-01-11T05:04:09.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.997+0000 7f6ea9d9b700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6e98031090 0x7f6e98033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:09.999 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:09.997+0000 7f6ea9d9b700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6e98031090 0x7f6e98033550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6e9c0096a0 tx=0x7f6e9c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:10.002 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.002+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6e9401ee60 con 0x7f6eac0f54b0 2024-01-11T05:04:10.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.208+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6eac1080a0 con 0x7f6eac0f54b0 2024-01-11T05:04:10.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.209+0000 7f6ea2ffd700 1 -- 172.21.15.169:0/3183017626 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e9401d380 con 0x7f6eac0f54b0 2024-01-11T05:04:10.210 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:10.210 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:10.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6e98031090 msgr2=0x7f6e98033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:10.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6e98031090 0x7f6e98033550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6e9c0096a0 tx=0x7f6e9c006b40).stop 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 msgr2=0x7f6eac106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac106840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f6e94009f80 tx=0x7f6e9400b040).stop 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 shutdown_connections 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6e98031090 0x7f6e98033550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 --2- 172.21.15.169:0/3183017626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6eac0f54b0 0x7f6eac106840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 >> 172.21.15.169:0/3183017626 conn(0x7f6eac0f0c10 msgr2=0x7f6eac05ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 shutdown_connections 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:10.211+0000 7f6eb092e700 1 -- 172.21.15.169:0/3183017626 wait complete. 2024-01-11T05:04:10.212 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:10.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:10 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:10.210306+0000 mon.smithi067 (mon.0) 116 : audit [DBG] from='client.? 172.21.15.169:0/3183017626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:11.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:11.601 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:10.274212+0000 mgr.smithi067.srbzpc (mgr.14156) 26 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:12.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/3952057707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 msgr2=0x7f9edc0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:12.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 --2- 172.21.15.169:0/3952057707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc0f56f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f9ed0009fb0 tx=0x7f9ed0009b30).stop 2024-01-11T05:04:12.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/3952057707 shutdown_connections 2024-01-11T05:04:12.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 --2- 172.21.15.169:0/3952057707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc0f56f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:12.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/3952057707 >> 172.21.15.169:0/3952057707 conn(0x7f9edc0f0a30 msgr2=0x7f9edc0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:12.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/3952057707 shutdown_connections 2024-01-11T05:04:12.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/3952057707 wait complete. 2024-01-11T05:04:12.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 Processor -- start 2024-01-11T05:04:12.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- start start 2024-01-11T05:04:12.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:12.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.764+0000 7f9ee1ca3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc060300 con 0x7f9edc0f52d0 2024-01-11T05:04:12.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.765+0000 7f9ee0ca1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:12.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.765+0000 7f9ee0ca1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39138/0 (socket says 172.21.15.169:39138) 2024-01-11T05:04:12.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.765+0000 7f9ee0ca1700 1 -- 172.21.15.169:0/18635727 learned_addr learned my addr 172.21.15.169:0/18635727 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:12.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.765+0000 7f9ee0ca1700 1 -- 172.21.15.169:0/18635727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed0005040 con 0x7f9edc0f52d0 2024-01-11T05:04:12.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.765+0000 7f9ee0ca1700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc109330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9ed0000c00 tx=0x7f9ed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:12.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.765+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ed00095f0 con 0x7f9edc0f52d0 2024-01-11T05:04:12.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc060500 con 0x7f9edc0f52d0 2024-01-11T05:04:12.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9ed0007e50 con 0x7f9edc0f52d0 2024-01-11T05:04:12.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ed0014e70 con 0x7f9edc0f52d0 2024-01-11T05:04:12.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc1099e0 con 0x7f9edc0f52d0 2024-01-11T05:04:12.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9ed0025070 con 0x7f9edc0f52d0 2024-01-11T05:04:12.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ed97fa700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9ecc031090 0x7f9ecc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:12.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.766+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc0591e0 con 0x7f9edc0f52d0 2024-01-11T05:04:12.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.767+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9ed0041d40 con 0x7f9edc0f52d0 2024-01-11T05:04:12.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.767+0000 7f9edbfff700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9ecc031090 0x7f9ecc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:12.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.767+0000 7f9edbfff700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9ecc031090 0x7f9ecc033550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9ec80096a0 tx=0x7f9ec8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:12.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.771+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9ed00079c0 con 0x7f9edc0f52d0 2024-01-11T05:04:12.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.978+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9edc045bb0 con 0x7f9edc0f52d0 2024-01-11T05:04:12.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.979+0000 7f9ed97fa700 1 -- 172.21.15.169:0/18635727 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ed000f300 con 0x7f9edc0f52d0 2024-01-11T05:04:12.980 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:12.980 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:12.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9ecc031090 msgr2=0x7f9ecc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:12.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9ecc031090 0x7f9ecc033550 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9ec80096a0 tx=0x7f9ec8006b40).stop 2024-01-11T05:04:12.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 msgr2=0x7f9edc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:12.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc109330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9ed0000c00 tx=0x7f9ed000b040).stop 2024-01-11T05:04:12.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 shutdown_connections 2024-01-11T05:04:12.981 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9ecc031090 0x7f9ecc033550 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:12.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 --2- 172.21.15.169:0/18635727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc0f52d0 0x7f9edc109330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:12.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 >> 172.21.15.169:0/18635727 conn(0x7f9edc0f0a30 msgr2=0x7f9edc0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:12.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 shutdown_connections 2024-01-11T05:04:12.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:12.980+0000 7f9ee1ca3700 1 -- 172.21.15.169:0/18635727 wait complete. 2024-01-11T05:04:12.982 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:12.274457+0000 mgr.smithi067.srbzpc (mgr.14156) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:12.980059+0000 mon.smithi067 (mon.0) 117 : audit [DBG] from='client.? 172.21.15.169:0/18635727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:14.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:14.361 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:15.261 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:14.274709+0000 mgr.smithi067.srbzpc (mgr.14156) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.509+0000 7fc17664e700 1 -- 172.21.15.169:0/2485606671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 msgr2=0x7fc1700f6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.509+0000 7fc17664e700 1 --2- 172.21.15.169:0/2485606671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc1700f6cf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc164009fb0 tx=0x7fc164009b30).stop 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.509+0000 7fc17664e700 1 -- 172.21.15.169:0/2485606671 shutdown_connections 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.509+0000 7fc17664e700 1 --2- 172.21.15.169:0/2485606671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc1700f6cf0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.509+0000 7fc17664e700 1 -- 172.21.15.169:0/2485606671 >> 172.21.15.169:0/2485606671 conn(0x7fc1700f01d0 msgr2=0x7fc1700f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.509+0000 7fc17664e700 1 -- 172.21.15.169:0/2485606671 shutdown_connections 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17664e700 1 -- 172.21.15.169:0/2485606671 wait complete. 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17664e700 1 Processor -- start 2024-01-11T05:04:15.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17664e700 1 -- start start 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17664e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc17010b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17664e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc170107120 con 0x7fc1700f68d0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17564c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc17010b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17564c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc17010b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39162/0 (socket says 172.21.15.169:39162) 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.510+0000 7fc17564c700 1 -- 172.21.15.169:0/4155436129 learned_addr learned my addr 172.21.15.169:0/4155436129 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc17564c700 1 -- 172.21.15.169:0/4155436129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164005040 con 0x7fc1700f68d0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc17564c700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc17010b730 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc1640089f0 tx=0x7fc16400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc164009320 con 0x7fc1700f68d0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1700f5240 con 0x7fc1700f68d0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc164013ec0 con 0x7fc1700f68d0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc164014d30 con 0x7fc1700f68d0 2024-01-11T05:04:15.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.511+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17010bc20 con 0x7fc1700f68d0 2024-01-11T05:04:15.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.512+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc1640263a0 con 0x7fc1700f68d0 2024-01-11T05:04:15.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.512+0000 7fc161ffb700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc15c031090 0x7fc15c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:15.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.512+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc1640424a0 con 0x7fc1700f68d0 2024-01-11T05:04:15.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.512+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc170045bb0 con 0x7fc1700f68d0 2024-01-11T05:04:15.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.512+0000 7fc174e4b700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc15c031090 0x7fc15c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:15.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.513+0000 7fc174e4b700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc15c031090 0x7fc15c033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fc16c0096a0 tx=0x7fc16c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:15.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.517+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc1640266d0 con 0x7fc1700f68d0 2024-01-11T05:04:15.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.724+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc17010c1d0 con 0x7fc1700f68d0 2024-01-11T05:04:15.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.725+0000 7fc161ffb700 1 -- 172.21.15.169:0/4155436129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc16402b020 con 0x7fc1700f68d0 2024-01-11T05:04:15.726 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:15.726 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc15c031090 msgr2=0x7fc15c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc15c031090 0x7fc15c033550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fc16c0096a0 tx=0x7fc16c006b40).stop 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 msgr2=0x7fc17010b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc17010b730 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc1640089f0 tx=0x7fc16400b040).stop 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 shutdown_connections 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc15c031090 0x7fc15c033550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 --2- 172.21.15.169:0/4155436129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1700f68d0 0x7fc17010b730 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 >> 172.21.15.169:0/4155436129 conn(0x7fc1700f01d0 msgr2=0x7fc1700f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 shutdown_connections 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:15.727+0000 7fc17664e700 1 -- 172.21.15.169:0/4155436129 wait complete. 2024-01-11T05:04:15.728 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:16.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:16 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:15.726119+0000 mon.smithi067 (mon.0) 118 : audit [DBG] from='client.? 172.21.15.169:0/4155436129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:17.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:17.110 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:17.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:16.274992+0000 mgr.smithi067.srbzpc (mgr.14156) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3148445949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 msgr2=0x7fd3f40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 --2- 172.21.15.169:0/3148445949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f40f69b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd3e8004760 tx=0x7fd3e8009ad0).stop 2024-01-11T05:04:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3148445949 shutdown_connections 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 --2- 172.21.15.169:0/3148445949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f40f69b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3148445949 >> 172.21.15.169:0/3148445949 conn(0x7fd3f40f1cd0 msgr2=0x7fd3f40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3148445949 shutdown_connections 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.281+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3148445949 wait complete. 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f88b7700 1 Processor -- start 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f88b7700 1 -- start start 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f88b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f88b7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4069f00 con 0x7fd3f40f6590 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39182/0 (socket says 172.21.15.169:39182) 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f259c700 1 -- 172.21.15.169:0/3523007672 learned_addr learned my addr 172.21.15.169:0/3523007672 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f259c700 1 -- 172.21.15.169:0/3523007672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8009d70 con 0x7fd3f40f6590 2024-01-11T05:04:18.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.282+0000 7fd3f259c700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f406ba80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fd3e8004170 tx=0x7fd3e8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:18.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.283+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e800e040 con 0x7fd3f40f6590 2024-01-11T05:04:18.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.283+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f406a0a0 con 0x7fd3f40f6590 2024-01-11T05:04:18.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.283+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd3e8008ef0 con 0x7fd3f40f6590 2024-01-11T05:04:18.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.283+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e8014c90 con 0x7fd3f40f6590 2024-01-11T05:04:18.285 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.283+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f406a530 con 0x7fd3f40f6590 2024-01-11T05:04:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.283+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd3e8014420 con 0x7fd3f40f6590 2024-01-11T05:04:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.284+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f40591e0 con 0x7fd3f40f6590 2024-01-11T05:04:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.284+0000 7fd3e2ffd700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3dc031090 0x7fd3dc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.284+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd3e80421f0 con 0x7fd3f40f6590 2024-01-11T05:04:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.284+0000 7fd3f1d9b700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3dc031090 0x7fd3dc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.284+0000 7fd3f1d9b700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3dc031090 0x7fd3dc033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fd3e40096a0 tx=0x7fd3e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:18.289 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.288+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd3e8026300 con 0x7fd3f40f6590 2024-01-11T05:04:18.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.498+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3f40fc080 con 0x7fd3f40f6590 2024-01-11T05:04:18.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.499+0000 7fd3e2ffd700 1 -- 172.21.15.169:0/3523007672 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3e8012b20 con 0x7fd3f40f6590 2024-01-11T05:04:18.499 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:18.499 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:18.501 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3dc031090 msgr2=0x7fd3dc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3dc031090 0x7fd3dc033550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fd3e40096a0 tx=0x7fd3e4006b40).stop 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 msgr2=0x7fd3f406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f406ba80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fd3e8004170 tx=0x7fd3e8009ad0).stop 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 shutdown_connections 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd3dc031090 0x7fd3dc033550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 --2- 172.21.15.169:0/3523007672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f40f6590 0x7fd3f406ba80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 >> 172.21.15.169:0/3523007672 conn(0x7fd3f40f1cd0 msgr2=0x7fd3f40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 shutdown_connections 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:18.500+0000 7fd3f88b7700 1 -- 172.21.15.169:0/3523007672 wait complete. 2024-01-11T05:04:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:18.275337+0000 mgr.smithi067.srbzpc (mgr.14156) 30 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:18.499925+0000 mon.smithi067 (mon.0) 119 : audit [DBG] from='client.? 172.21.15.169:0/3523007672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:19.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:19.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 -- 172.21.15.169:0/2994103473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 msgr2=0x7f39540f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 --2- 172.21.15.169:0/2994103473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f39540f58d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3944009fb0 tx=0x7f3944009b30).stop 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 -- 172.21.15.169:0/2994103473 shutdown_connections 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 --2- 172.21.15.169:0/2994103473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f39540f58d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 -- 172.21.15.169:0/2994103473 >> 172.21.15.169:0/2994103473 conn(0x7f39540f0c10 msgr2=0x7f39540f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 -- 172.21.15.169:0/2994103473 shutdown_connections 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.955+0000 7f395a941700 1 -- 172.21.15.169:0/2994103473 wait complete. 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f395a941700 1 Processor -- start 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f395a941700 1 -- start start 2024-01-11T05:04:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f395a941700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f395410b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f395a941700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3954107310 con 0x7f39540f54b0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f3953fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f395410b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f3953fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f395410b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36282/0 (socket says 172.21.15.169:36282) 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.956+0000 7f3953fff700 1 -- 172.21.15.169:0/289087817 learned_addr learned my addr 172.21.15.169:0/289087817 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f3953fff700 1 -- 172.21.15.169:0/289087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944005040 con 0x7f39540f54b0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f3953fff700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f395410b920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f39440089f0 tx=0x7f394400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3944009320 con 0x7f39540f54b0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3954060300 con 0x7f39540f54b0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3944013ec0 con 0x7f39540f54b0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3944014d30 con 0x7f39540f54b0 2024-01-11T05:04:20.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.957+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395410be10 con 0x7f39540f54b0 2024-01-11T05:04:20.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.958+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3944014420 con 0x7f39540f54b0 2024-01-11T05:04:20.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.958+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3954045bb0 con 0x7f39540f54b0 2024-01-11T05:04:20.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.958+0000 7f3950ff9700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f393c031090 0x7f393c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:20.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.958+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f39440424c0 con 0x7f39540f54b0 2024-01-11T05:04:20.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.958+0000 7f39537fe700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f393c031090 0x7f393c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:20.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.959+0000 7f39537fe700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f393c031090 0x7f393c033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f39480096a0 tx=0x7f3948006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:20.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:20.963+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f394401d980 con 0x7f39540f54b0 2024-01-11T05:04:21.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.169+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f395410c5d0 con 0x7f39540f54b0 2024-01-11T05:04:21.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.169+0000 7f3950ff9700 1 -- 172.21.15.169:0/289087817 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f394401d380 con 0x7f39540f54b0 2024-01-11T05:04:21.170 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:21.170 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f393c031090 msgr2=0x7f393c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f393c031090 0x7f393c033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f39480096a0 tx=0x7f3948006b40).stop 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 msgr2=0x7f395410b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f395410b920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f39440089f0 tx=0x7f394400b040).stop 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 shutdown_connections 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f393c031090 0x7f393c033550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 --2- 172.21.15.169:0/289087817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39540f54b0 0x7f395410b920 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 >> 172.21.15.169:0/289087817 conn(0x7f39540f0c10 msgr2=0x7f39540f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 shutdown_connections 2024-01-11T05:04:21.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:21.171+0000 7f395a941700 1 -- 172.21.15.169:0/289087817 wait complete. 2024-01-11T05:04:21.173 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:20.275722+0000 mgr.smithi067.srbzpc (mgr.14156) 31 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:21.170653+0000 mon.smithi067 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.169:0/289087817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:22.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:22.559 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.728+0000 7fe22f095700 1 -- 172.21.15.169:0/3133740737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2280f6590 msgr2=0x7fe2280f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.728+0000 7fe22f095700 1 --2- 172.21.15.169:0/3133740737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2280f6590 0x7fe2280f69b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe21c004760 tx=0x7fe21c009ad0).stop 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.728+0000 7fe22f095700 1 -- 172.21.15.169:0/3133740737 shutdown_connections 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.728+0000 7fe22f095700 1 --2- 172.21.15.169:0/3133740737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2280f6590 0x7fe2280f69b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.728+0000 7fe22f095700 1 -- 172.21.15.169:0/3133740737 >> 172.21.15.169:0/3133740737 conn(0x7fe2280f1cd0 msgr2=0x7fe2280f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.729+0000 7fe22f095700 1 -- 172.21.15.169:0/3133740737 shutdown_connections 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.729+0000 7fe22f095700 1 -- 172.21.15.169:0/3133740737 wait complete. 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.729+0000 7fe22f095700 1 Processor -- start 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.729+0000 7fe22f095700 1 -- start start 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.729+0000 7fe22f095700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 0x7fe228106d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:23.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.729+0000 7fe22f095700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe228107200 con 0x7fe2281068f0 2024-01-11T05:04:23.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22ce31700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 0x7fe228106d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:23.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22ce31700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 0x7fe228106d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36290/0 (socket says 172.21.15.169:36290) 2024-01-11T05:04:23.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22ce31700 1 -- 172.21.15.169:0/2333018077 learned_addr learned my addr 172.21.15.169:0/2333018077 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:23.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22ce31700 1 -- 172.21.15.169:0/2333018077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c009d70 con 0x7fe2281068f0 2024-01-11T05:04:23.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22ce31700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 0x7fe228106d10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe21c009820 tx=0x7fe21c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe21c00e040 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe228107400 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe21c0095b0 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe21c014be0 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.730+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe228107f50 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.731+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe21c014420 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.731+0000 7fe2257fa700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe218031090 0x7fe218033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.731+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2280591e0 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.731+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe21c042160 con 0x7fe2281068f0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.731+0000 7fe227fff700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe218031090 0x7fe218033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:23.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.732+0000 7fe227fff700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe218031090 0x7fe218033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:23.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.736+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe21c01eb20 con 0x7fe2281068f0 2024-01-11T05:04:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:22.276031+0000 mgr.smithi067.srbzpc (mgr.14156) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:22.411822+0000 mon.smithi067 (mon.0) 121 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:04:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:22.412975+0000 mon.smithi067 (mon.0) 122 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:04:23.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.943+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe228108710 con 0x7fe2281068f0 2024-01-11T05:04:23.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.943+0000 7fe2257fa700 1 -- 172.21.15.169:0/2333018077 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe21c01eb20 con 0x7fe2281068f0 2024-01-11T05:04:23.944 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:23.944 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe218031090 msgr2=0x7fe218033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe218031090 0x7fe218033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).stop 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 msgr2=0x7fe228106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 0x7fe228106d10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe21c009820 tx=0x7fe21c009ad0).stop 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 shutdown_connections 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe218031090 0x7fe218033550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 --2- 172.21.15.169:0/2333018077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2281068f0 0x7fe228106d10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 >> 172.21.15.169:0/2333018077 conn(0x7fe2280f1cd0 msgr2=0x7fe2280f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 shutdown_connections 2024-01-11T05:04:23.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:23.945+0000 7fe22f095700 1 -- 172.21.15.169:0/2333018077 wait complete. 2024-01-11T05:04:23.947 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:24.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:24 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:23.944515+0000 mon.smithi067 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.169:0/2333018077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:25.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:25.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:25.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:24.276338+0000 mgr.smithi067.srbzpc (mgr.14156) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:26.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 -- 172.21.15.169:0/159108667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 msgr2=0x7efe740f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 --2- 172.21.15.169:0/159108667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe740f67f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7efe68009fb0 tx=0x7efe68009b30).stop 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 -- 172.21.15.169:0/159108667 shutdown_connections 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 --2- 172.21.15.169:0/159108667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe740f67f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 -- 172.21.15.169:0/159108667 >> 172.21.15.169:0/159108667 conn(0x7efe740f1b50 msgr2=0x7efe740f3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 -- 172.21.15.169:0/159108667 shutdown_connections 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 -- 172.21.15.169:0/159108667 wait complete. 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 Processor -- start 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.523+0000 7efe797fa700 1 -- start start 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe797fa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe7406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe797fa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe74069f00 con 0x7efe740f63d0 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe73fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe7406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe73fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe7406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36302/0 (socket says 172.21.15.169:36302) 2024-01-11T05:04:26.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe73fff700 1 -- 172.21.15.169:0/2225875458 learned_addr learned my addr 172.21.15.169:0/2225875458 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe73fff700 1 -- 172.21.15.169:0/2225875458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68005040 con 0x7efe740f63d0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.524+0000 7efe73fff700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe7406ba80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efe680089f0 tx=0x7efe6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.525+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe68009320 con 0x7efe740f63d0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.525+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7406a0a0 con 0x7efe740f63d0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.525+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efe68013ec0 con 0x7efe740f63d0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.525+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe68014d30 con 0x7efe740f63d0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.525+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7406a530 con 0x7efe740f63d0 2024-01-11T05:04:26.527 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.525+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efe680095f0 con 0x7efe740f63d0 2024-01-11T05:04:26.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.526+0000 7efe70ff9700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efe5c031090 0x7efe5c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:26.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.526+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe740591e0 con 0x7efe740f63d0 2024-01-11T05:04:26.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.526+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efe680424a0 con 0x7efe740f63d0 2024-01-11T05:04:26.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.526+0000 7efe737fe700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efe5c031090 0x7efe5c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:26.528 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.526+0000 7efe737fe700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efe5c031090 0x7efe5c033550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efe640096a0 tx=0x7efe64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:26.531 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.531+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efe68015040 con 0x7efe740f63d0 2024-01-11T05:04:26.743 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.743+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe740fbec0 con 0x7efe740f63d0 2024-01-11T05:04:26.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.744+0000 7efe70ff9700 1 -- 172.21.15.169:0/2225875458 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe68015040 con 0x7efe740f63d0 2024-01-11T05:04:26.744 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:26.745 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efe5c031090 msgr2=0x7efe5c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efe5c031090 0x7efe5c033550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efe640096a0 tx=0x7efe64006b40).stop 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 msgr2=0x7efe7406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe7406ba80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efe680089f0 tx=0x7efe6800b040).stop 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 shutdown_connections 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efe5c031090 0x7efe5c033550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:26.746 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 --2- 172.21.15.169:0/2225875458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe740f63d0 0x7efe7406ba80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:26.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 >> 172.21.15.169:0/2225875458 conn(0x7efe740f1b50 msgr2=0x7efe740f2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:26.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 shutdown_connections 2024-01-11T05:04:26.747 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:26.745+0000 7efe797fa700 1 -- 172.21.15.169:0/2225875458 wait complete. 2024-01-11T05:04:26.747 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:26.276726+0000 mgr.smithi067.srbzpc (mgr.14156) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:26.744936+0000 mon.smithi067 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.169:0/2225875458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:28.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:28.133 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:29.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.311+0000 7f44738b4700 1 -- 172.21.15.169:0/3809220962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 msgr2=0x7f446c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.311+0000 7f44738b4700 1 --2- 172.21.15.169:0/3809220962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c0f69b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f445c004760 tx=0x7f445c009ad0).stop 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.311+0000 7f44738b4700 1 -- 172.21.15.169:0/3809220962 shutdown_connections 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.311+0000 7f44738b4700 1 --2- 172.21.15.169:0/3809220962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c0f69b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.311+0000 7f44738b4700 1 -- 172.21.15.169:0/3809220962 >> 172.21.15.169:0/3809220962 conn(0x7f446c0f1cd0 msgr2=0x7f446c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.312+0000 7f44738b4700 1 -- 172.21.15.169:0/3809220962 shutdown_connections 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.312+0000 7f44738b4700 1 -- 172.21.15.169:0/3809220962 wait complete. 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.312+0000 7f44738b4700 1 Processor -- start 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.312+0000 7f44738b4700 1 -- start start 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.312+0000 7f44738b4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.312+0000 7f44738b4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c106cb0 con 0x7f446c0f6590 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4471650700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4471650700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36308/0 (socket says 172.21.15.169:36308) 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4471650700 1 -- 172.21.15.169:0/3850167123 learned_addr learned my addr 172.21.15.169:0/3850167123 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:29.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4471650700 1 -- 172.21.15.169:0/3850167123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c009d70 con 0x7f446c0f6590 2024-01-11T05:04:29.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4471650700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c1067c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f445c004170 tx=0x7f445c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f445c00e040 con 0x7f446c0f6590 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c106eb0 con 0x7f446c0f6590 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f445c008ef0 con 0x7f446c0f6590 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f445c014c90 con 0x7f446c0f6590 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.313+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c107340 con 0x7f446c0f6590 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.314+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c045bb0 con 0x7f446c0f6590 2024-01-11T05:04:29.316 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.314+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f445c014420 con 0x7f446c0f6590 2024-01-11T05:04:29.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.315+0000 7f4461ffb700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4458031090 0x7f4458033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:29.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.315+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f445c042260 con 0x7f446c0f6590 2024-01-11T05:04:29.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.315+0000 7f4470e4f700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4458031090 0x7f4458033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:29.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.315+0000 7f4470e4f700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4458031090 0x7f4458033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f44680096a0 tx=0x7f4468006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:29.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.319+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f445c026300 con 0x7f446c0f6590 2024-01-11T05:04:29.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.529+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f446c108190 con 0x7f446c0f6590 2024-01-11T05:04:29.530 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.530+0000 7f4461ffb700 1 -- 172.21.15.169:0/3850167123 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f445c012b20 con 0x7f446c0f6590 2024-01-11T05:04:29.531 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:29.531 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:29.532 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4458031090 msgr2=0x7f4458033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4458031090 0x7f4458033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f44680096a0 tx=0x7f4468006b40).stop 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 msgr2=0x7f446c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c1067c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f445c004170 tx=0x7f445c009ad0).stop 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 shutdown_connections 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4458031090 0x7f4458033550 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 --2- 172.21.15.169:0/3850167123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f446c0f6590 0x7f446c1067c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 >> 172.21.15.169:0/3850167123 conn(0x7f446c0f1cd0 msgr2=0x7f446c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 shutdown_connections 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:29.532+0000 7f44738b4700 1 -- 172.21.15.169:0/3850167123 wait complete. 2024-01-11T05:04:29.533 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:28.277221+0000 mgr.smithi067.srbzpc (mgr.14156) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:30.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:30 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:29.531183+0000 mon.smithi067 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.169:0/3850167123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:31.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:31.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:30.277521+0000 mgr.smithi067.srbzpc (mgr.14156) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/618457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 msgr2=0x7fd4c00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 --2- 172.21.15.169:0/618457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c00f56f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fd4b0009fb0 tx=0x7fd4b0009b30).stop 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/618457689 shutdown_connections 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 --2- 172.21.15.169:0/618457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c00f56f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/618457689 >> 172.21.15.169:0/618457689 conn(0x7fd4c00f0a30 msgr2=0x7fd4c00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/618457689 shutdown_connections 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/618457689 wait complete. 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 Processor -- start 2024-01-11T05:04:32.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.196+0000 7fd4c4dd1700 1 -- start start 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4c4dd1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c0106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4c4dd1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0106b10 con 0x7fd4c00f52d0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4bf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c0106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4bf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c0106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35826/0 (socket says 172.21.15.169:35826) 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4bf7fe700 1 -- 172.21.15.169:0/3875751518 learned_addr learned my addr 172.21.15.169:0/3875751518 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4bf7fe700 1 -- 172.21.15.169:0/3875751518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b0005040 con 0x7fd4c00f52d0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.197+0000 7fd4bf7fe700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c0106620 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fd4b00089f0 tx=0x7fd4b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.198+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4b0009320 con 0x7fd4c00f52d0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.198+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd4b0013ec0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.198+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c0106d10 con 0x7fd4c00f52d0 2024-01-11T05:04:32.199 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.198+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4b0014d30 con 0x7fd4c00f52d0 2024-01-11T05:04:32.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.198+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c01071a0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.200 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.199+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd4b00263a0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.199+0000 7fd4a7fff700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd4a8031090 0x7fd4a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:32.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.199+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c0045bb0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.199+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd4b00423b0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.199+0000 7fd4beffd700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd4a8031090 0x7fd4a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:32.201 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.199+0000 7fd4beffd700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd4a8031090 0x7fd4a8033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd4b40096a0 tx=0x7fd4b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:32.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.203+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd4b00266d0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.411+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4c0107fa0 con 0x7fd4c00f52d0 2024-01-11T05:04:32.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.412+0000 7fd4a7fff700 1 -- 172.21.15.169:0/3875751518 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4b002b020 con 0x7fd4c00f52d0 2024-01-11T05:04:32.413 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:32.413 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd4a8031090 msgr2=0x7fd4a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd4a8031090 0x7fd4a8033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd4b40096a0 tx=0x7fd4b4006b40).stop 2024-01-11T05:04:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 msgr2=0x7fd4c0106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c0106620 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fd4b00089f0 tx=0x7fd4b000b040).stop 2024-01-11T05:04:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 shutdown_connections 2024-01-11T05:04:32.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd4a8031090 0x7fd4a8033550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:32.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 --2- 172.21.15.169:0/3875751518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c00f52d0 0x7fd4c0106620 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:32.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.413+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 >> 172.21.15.169:0/3875751518 conn(0x7fd4c00f0a30 msgr2=0x7fd4c005ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:32.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.414+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 shutdown_connections 2024-01-11T05:04:32.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:32.414+0000 7fd4c4dd1700 1 -- 172.21.15.169:0/3875751518 wait complete. 2024-01-11T05:04:32.415 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:33.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:33.736 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:32.277879+0000 mgr.smithi067.srbzpc (mgr.14156) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:32.413241+0000 mon.smithi067 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.169:0/3875751518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:34.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4130703803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 msgr2=0x7fb6f00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:34.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 --2- 172.21.15.169:0/4130703803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f00f58a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb6e4009fb0 tx=0x7fb6e4009b30).stop 2024-01-11T05:04:34.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4130703803 shutdown_connections 2024-01-11T05:04:34.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 --2- 172.21.15.169:0/4130703803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f00f58a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:34.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4130703803 >> 172.21.15.169:0/4130703803 conn(0x7fb6f00f0be0 msgr2=0x7fb6f00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:34.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4130703803 shutdown_connections 2024-01-11T05:04:34.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.897+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4130703803 wait complete. 2024-01-11T05:04:34.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6f4f90700 1 Processor -- start 2024-01-11T05:04:34.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6f4f90700 1 -- start start 2024-01-11T05:04:34.901 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6f4f90700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:34.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6f4f90700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0105be0 con 0x7fb6f00f5480 2024-01-11T05:04:34.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6ee59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:34.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6ee59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35832/0 (socket says 172.21.15.169:35832) 2024-01-11T05:04:34.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.898+0000 7fb6ee59c700 1 -- 172.21.15.169:0/4156761204 learned_addr learned my addr 172.21.15.169:0/4156761204 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:34.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6ee59c700 1 -- 172.21.15.169:0/4156761204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4005040 con 0x7fb6f00f5480 2024-01-11T05:04:34.902 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6ee59c700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f01056f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb6e40089f0 tx=0x7fb6e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:34.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e4009320 con 0x7fb6f00f5480 2024-01-11T05:04:34.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f0105de0 con 0x7fb6f00f5480 2024-01-11T05:04:34.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb6e4013ec0 con 0x7fb6f00f5480 2024-01-11T05:04:34.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6e4014d30 con 0x7fb6f00f5480 2024-01-11T05:04:34.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.899+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f0106270 con 0x7fb6f00f5480 2024-01-11T05:04:34.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.900+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb6e40095f0 con 0x7fb6f00f5480 2024-01-11T05:04:34.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.900+0000 7fb6deffd700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb6d8031090 0x7fb6d8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:34.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.900+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb6e40423e0 con 0x7fb6f00f5480 2024-01-11T05:04:34.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.900+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f0045bb0 con 0x7fb6f00f5480 2024-01-11T05:04:34.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.900+0000 7fb6edd9b700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb6d8031090 0x7fb6d8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:34.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.901+0000 7fb6edd9b700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb6d8031090 0x7fb6d8033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb6e00096a0 tx=0x7fb6e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:34.906 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:34.905+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb6e401dd70 con 0x7fb6f00f5480 2024-01-11T05:04:35.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.115+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6f00faf70 con 0x7fb6f00f5480 2024-01-11T05:04:35.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.115+0000 7fb6deffd700 1 -- 172.21.15.169:0/4156761204 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6f00faf70 con 0x7fb6f00f5480 2024-01-11T05:04:35.117 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:35.117 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:35.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb6d8031090 msgr2=0x7fb6d8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:35.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb6d8031090 0x7fb6d8033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb6e00096a0 tx=0x7fb6e0006b40).stop 2024-01-11T05:04:35.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 msgr2=0x7fb6f01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:35.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f01056f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb6e40089f0 tx=0x7fb6e400b040).stop 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 shutdown_connections 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb6d8031090 0x7fb6d8033550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 --2- 172.21.15.169:0/4156761204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f00f5480 0x7fb6f01056f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 >> 172.21.15.169:0/4156761204 conn(0x7fb6f00f0be0 msgr2=0x7fb6f00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 shutdown_connections 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:35.117+0000 7fb6f4f90700 1 -- 172.21.15.169:0/4156761204 wait complete. 2024-01-11T05:04:35.119 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:35.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:34.278284+0000 mgr.smithi067.srbzpc (mgr.14156) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:36.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:35.116863+0000 mon.smithi067 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.169:0/4156761204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:36.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:36.517 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:37.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:36.278635+0000 mgr.smithi067.srbzpc (mgr.14156) 39 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 -- 172.21.15.169:0/3543429670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 msgr2=0x7f08f00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 --2- 172.21.15.169:0/3543429670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f00f69b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f08ec004760 tx=0x7f08ec009ad0).stop 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 -- 172.21.15.169:0/3543429670 shutdown_connections 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 --2- 172.21.15.169:0/3543429670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f00f69b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 -- 172.21.15.169:0/3543429670 >> 172.21.15.169:0/3543429670 conn(0x7f08f00f1cd0 msgr2=0x7f08f00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 -- 172.21.15.169:0/3543429670 shutdown_connections 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.668+0000 7f08f7a65700 1 -- 172.21.15.169:0/3543429670 wait complete. 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f7a65700 1 Processor -- start 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f7a65700 1 -- start start 2024-01-11T05:04:37.671 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f7a65700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f7a65700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f0106cb0 con 0x7f08f00f6590 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f5801700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f5801700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35848/0 (socket says 172.21.15.169:35848) 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.669+0000 7f08f5801700 1 -- 172.21.15.169:0/448768896 learned_addr learned my addr 172.21.15.169:0/448768896 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08f5801700 1 -- 172.21.15.169:0/448768896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec009d70 con 0x7f08f00f6590 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08f5801700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f01067c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f08ec004170 tx=0x7f08ec009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08ec00e040 con 0x7f08f00f6590 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f0106eb0 con 0x7f08f00f6590 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f08ec008ef0 con 0x7f08f00f6590 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08ec014c90 con 0x7f08f00f6590 2024-01-11T05:04:37.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.670+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f0107340 con 0x7f08f00f6590 2024-01-11T05:04:37.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.671+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f08ec014420 con 0x7f08f00f6590 2024-01-11T05:04:37.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.671+0000 7f08e5ffb700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f08dc031090 0x7f08dc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:37.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.671+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f08ec042240 con 0x7f08f00f6590 2024-01-11T05:04:37.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.671+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f0045bb0 con 0x7f08f00f6590 2024-01-11T05:04:37.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.671+0000 7f08f5000700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f08dc031090 0x7f08dc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:37.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.672+0000 7f08f5000700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f08dc031090 0x7f08dc033550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f08e00096a0 tx=0x7f08e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:37.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.676+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f08ec012d00 con 0x7f08f00f6590 2024-01-11T05:04:37.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.888+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08f00fc170 con 0x7f08f00f6590 2024-01-11T05:04:37.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.889+0000 7f08e5ffb700 1 -- 172.21.15.169:0/448768896 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f08ec012b20 con 0x7f08f00f6590 2024-01-11T05:04:37.889 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:37.890 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:37.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f08dc031090 msgr2=0x7f08dc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:37.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f08dc031090 0x7f08dc033550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f08e00096a0 tx=0x7f08e0006b40).stop 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 msgr2=0x7f08f01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f01067c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f08ec004170 tx=0x7f08ec009ad0).stop 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 shutdown_connections 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f08dc031090 0x7f08dc033550 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 --2- 172.21.15.169:0/448768896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08f00f6590 0x7f08f01067c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 >> 172.21.15.169:0/448768896 conn(0x7f08f00f1cd0 msgr2=0x7f08f00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 shutdown_connections 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:37.890+0000 7f08f7a65700 1 -- 172.21.15.169:0/448768896 wait complete. 2024-01-11T05:04:37.892 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:38.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:37.890175+0000 mon.smithi067 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.169:0/448768896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:39.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:39.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:39.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:38.279069+0000 mgr.smithi067.srbzpc (mgr.14156) 40 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:40.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 -- 172.21.15.169:0/2511682283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 msgr2=0x7f7e980f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 --2- 172.21.15.169:0/2511682283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e980f58a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7e88009fb0 tx=0x7f7e88009b30).stop 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 -- 172.21.15.169:0/2511682283 shutdown_connections 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 --2- 172.21.15.169:0/2511682283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e980f58a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 -- 172.21.15.169:0/2511682283 >> 172.21.15.169:0/2511682283 conn(0x7f7e980f0be0 msgr2=0x7f7e980f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 -- 172.21.15.169:0/2511682283 shutdown_connections 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.394+0000 7f7e9cef3700 1 -- 172.21.15.169:0/2511682283 wait complete. 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9cef3700 1 Processor -- start 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9cef3700 1 -- start start 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9cef3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e981056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9cef3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98105be0 con 0x7f7e980f5480 2024-01-11T05:04:40.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e981056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e981056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60662/0 (socket says 172.21.15.169:60662) 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.395+0000 7f7e9659c700 1 -- 172.21.15.169:0/400636662 learned_addr learned my addr 172.21.15.169:0/400636662 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e9659c700 1 -- 172.21.15.169:0/400636662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e88005040 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e9659c700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e981056f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7e880089f0 tx=0x7f7e8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e88009320 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e98105de0 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7e88013ec0 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e88014d30 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.396+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e98106270 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.397+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7e88014420 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.397+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e98045bb0 con 0x7f7e980f5480 2024-01-11T05:04:40.399 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.397+0000 7f7e8effd700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7e84031090 0x7f7e84033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:40.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.397+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7e880423e0 con 0x7f7e980f5480 2024-01-11T05:04:40.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.397+0000 7f7e95d9b700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7e84031090 0x7f7e84033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:40.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.398+0000 7f7e95d9b700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7e84031090 0x7f7e84033550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7e800096a0 tx=0x7f7e80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:40.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.402+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7e880268c0 con 0x7f7e980f5480 2024-01-11T05:04:40.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.610+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e980faf70 con 0x7f7e980f5480 2024-01-11T05:04:40.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.611+0000 7f7e8effd700 1 -- 172.21.15.169:0/400636662 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e8801d7f0 con 0x7f7e980f5480 2024-01-11T05:04:40.612 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:40.612 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:40.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7e84031090 msgr2=0x7f7e84033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:40.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7e84031090 0x7f7e84033550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7e800096a0 tx=0x7f7e80006b40).stop 2024-01-11T05:04:40.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 msgr2=0x7f7e981056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e981056f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7e880089f0 tx=0x7f7e8800b040).stop 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 shutdown_connections 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7e84031090 0x7f7e84033550 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 --2- 172.21.15.169:0/400636662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e980f5480 0x7f7e981056f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 >> 172.21.15.169:0/400636662 conn(0x7f7e980f0be0 msgr2=0x7f7e980f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 shutdown_connections 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:40.613+0000 7f7e9cef3700 1 -- 172.21.15.169:0/400636662 wait complete. 2024-01-11T05:04:40.614 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:40.279513+0000 mgr.smithi067.srbzpc (mgr.14156) 41 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:40.612262+0000 mon.smithi067 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.169:0/400636662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:41.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:41.995 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:43.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 -- 172.21.15.169:0/3153201562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 msgr2=0x7f4ea00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 --2- 172.21.15.169:0/3153201562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea00f58a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4e94009fb0 tx=0x7f4e94009b30).stop 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 -- 172.21.15.169:0/3153201562 shutdown_connections 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 --2- 172.21.15.169:0/3153201562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea00f58a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 -- 172.21.15.169:0/3153201562 >> 172.21.15.169:0/3153201562 conn(0x7f4ea00f0be0 msgr2=0x7f4ea00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 -- 172.21.15.169:0/3153201562 shutdown_connections 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.108+0000 7f4ea7220700 1 -- 172.21.15.169:0/3153201562 wait complete. 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.109+0000 7f4ea7220700 1 Processor -- start 2024-01-11T05:04:43.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.109+0000 7f4ea7220700 1 -- start start 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.109+0000 7f4ea7220700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea01094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.109+0000 7f4ea7220700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea0060320 con 0x7f4ea00f5480 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea4fbc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea01094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea4fbc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea01094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60676/0 (socket says 172.21.15.169:60676) 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea4fbc700 1 -- 172.21.15.169:0/1957763754 learned_addr learned my addr 172.21.15.169:0/1957763754 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea4fbc700 1 -- 172.21.15.169:0/1957763754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94005040 con 0x7f4ea00f5480 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea4fbc700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea01094f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f4e940089f0 tx=0x7f4e9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:43.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e94009320 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea0060520 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4e94013ec0 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e94014d30 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.110+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea0109ba0 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.111+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4e940263a0 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.111+0000 7f4e9d7fa700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e90031090 0x7f4e90033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.111+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4e94042470 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.111+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea00591e0 con 0x7f4ea00f5480 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.112+0000 7f4e9ffff700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e90031090 0x7f4e90033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:43.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.112+0000 7f4e9ffff700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e90031090 0x7f4e90033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f4e8c0096a0 tx=0x7f4e8c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:43.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.116+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4e940266d0 con 0x7f4ea00f5480 2024-01-11T05:04:43.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.326+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ea00faf70 con 0x7f4ea00f5480 2024-01-11T05:04:43.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.327+0000 7f4e9d7fa700 1 -- 172.21.15.169:0/1957763754 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e94012330 con 0x7f4ea00f5480 2024-01-11T05:04:43.327 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:43.327 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:43.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e90031090 msgr2=0x7f4e90033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:43.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e90031090 0x7f4e90033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f4e8c0096a0 tx=0x7f4e8c006b40).stop 2024-01-11T05:04:43.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 msgr2=0x7f4ea01094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:43.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea01094f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f4e940089f0 tx=0x7f4e9400b040).stop 2024-01-11T05:04:43.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 shutdown_connections 2024-01-11T05:04:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e90031090 0x7f4e90033550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 --2- 172.21.15.169:0/1957763754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00f5480 0x7f4ea01094f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.328+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 >> 172.21.15.169:0/1957763754 conn(0x7f4ea00f0be0 msgr2=0x7f4ea00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.329+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 shutdown_connections 2024-01-11T05:04:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:43.329+0000 7f4ea7220700 1 -- 172.21.15.169:0/1957763754 wait complete. 2024-01-11T05:04:43.330 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:42.279873+0000 mgr.smithi067.srbzpc (mgr.14156) 42 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:43.327905+0000 mon.smithi067 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.169:0/1957763754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:44.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:44.702 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:44.280191+0000 mgr.smithi067.srbzpc (mgr.14156) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 -- 172.21.15.169:0/2944915287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 msgr2=0x7f6d580f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 --2- 172.21.15.169:0/2944915287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d580f58a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f6d48009fb0 tx=0x7f6d48009b30).stop 2024-01-11T05:04:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 -- 172.21.15.169:0/2944915287 shutdown_connections 2024-01-11T05:04:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 --2- 172.21.15.169:0/2944915287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d580f58a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 -- 172.21.15.169:0/2944915287 >> 172.21.15.169:0/2944915287 conn(0x7f6d580f0be0 msgr2=0x7f6d580f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:45.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 -- 172.21.15.169:0/2944915287 shutdown_connections 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.826+0000 7f6d60777700 1 -- 172.21.15.169:0/2944915287 wait complete. 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d60777700 1 Processor -- start 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d60777700 1 -- start start 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d60777700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d581056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d60777700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d58105be0 con 0x7f6d580f5480 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d5e513700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d581056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d5e513700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d581056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60704/0 (socket says 172.21.15.169:60704) 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.827+0000 7f6d5e513700 1 -- 172.21.15.169:0/1582871150 learned_addr learned my addr 172.21.15.169:0/1582871150 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d5e513700 1 -- 172.21.15.169:0/1582871150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48005040 con 0x7f6d580f5480 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d5e513700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d581056f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6d480089f0 tx=0x7f6d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d48009320 con 0x7f6d580f5480 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d58105de0 con 0x7f6d580f5480 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6d48013ec0 con 0x7f6d580f5480 2024-01-11T05:04:45.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d48014d30 con 0x7f6d580f5480 2024-01-11T05:04:45.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.828+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d58106270 con 0x7f6d580f5480 2024-01-11T05:04:45.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.829+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6d480095f0 con 0x7f6d580f5480 2024-01-11T05:04:45.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.829+0000 7f6d4effd700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d44031090 0x7f6d44033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:45.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.829+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6d480423e0 con 0x7f6d580f5480 2024-01-11T05:04:45.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.829+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d58045bb0 con 0x7f6d580f5480 2024-01-11T05:04:45.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.829+0000 7f6d5dd12700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d44031090 0x7f6d44033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:45.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.830+0000 7f6d5dd12700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d44031090 0x7f6d44033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6d540096a0 tx=0x7f6d54006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:45.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:45.834+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6d4801dd70 con 0x7f6d580f5480 2024-01-11T05:04:46.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.044+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d58106f20 con 0x7f6d580f5480 2024-01-11T05:04:46.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.045+0000 7f6d4effd700 1 -- 172.21.15.169:0/1582871150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d4802b020 con 0x7f6d580f5480 2024-01-11T05:04:46.046 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:46.046 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:46.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d44031090 msgr2=0x7f6d44033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:46.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d44031090 0x7f6d44033550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6d540096a0 tx=0x7f6d54006b40).stop 2024-01-11T05:04:46.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 msgr2=0x7f6d581056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:46.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d581056f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6d480089f0 tx=0x7f6d4800b040).stop 2024-01-11T05:04:46.047 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 shutdown_connections 2024-01-11T05:04:46.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d44031090 0x7f6d44033550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:46.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 --2- 172.21.15.169:0/1582871150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d580f5480 0x7f6d581056f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:46.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 >> 172.21.15.169:0/1582871150 conn(0x7f6d580f0be0 msgr2=0x7f6d580f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:46.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 shutdown_connections 2024-01-11T05:04:46.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:46.046+0000 7f6d60777700 1 -- 172.21.15.169:0/1582871150 wait complete. 2024-01-11T05:04:46.048 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:46.045892+0000 mon.smithi067 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.169:0/1582871150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:47.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:47.422 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:47.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:46.280599+0000 mgr.smithi067.srbzpc (mgr.14156) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:48.585 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 -- 172.21.15.169:0/1135385542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 msgr2=0x7f230c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 --2- 172.21.15.169:0/1135385542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c0f58d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2308009fb0 tx=0x7f2308009b30).stop 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 -- 172.21.15.169:0/1135385542 shutdown_connections 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 --2- 172.21.15.169:0/1135385542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c0f58d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 -- 172.21.15.169:0/1135385542 >> 172.21.15.169:0/1135385542 conn(0x7f230c0f0c10 msgr2=0x7f230c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 -- 172.21.15.169:0/1135385542 shutdown_connections 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.583+0000 7f23138ee700 1 -- 172.21.15.169:0/1135385542 wait complete. 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f23138ee700 1 Processor -- start 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f23138ee700 1 -- start start 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f23138ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c1056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:48.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f23138ee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c105b90 con 0x7f230c0f54b0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f231168a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c1056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f231168a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c1056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60718/0 (socket says 172.21.15.169:60718) 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f231168a700 1 -- 172.21.15.169:0/3222225813 learned_addr learned my addr 172.21.15.169:0/3222225813 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f231168a700 1 -- 172.21.15.169:0/3222225813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308005040 con 0x7f230c0f54b0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.584+0000 7f231168a700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c1056a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2308000c00 tx=0x7f230800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.585+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23080095f0 con 0x7f230c0f54b0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.585+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c105d90 con 0x7f230c0f54b0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.585+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2308007e50 con 0x7f230c0f54b0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.585+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2308014e70 con 0x7f230c0f54b0 2024-01-11T05:04:48.587 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.585+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c106220 con 0x7f230c0f54b0 2024-01-11T05:04:48.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.586+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2308025070 con 0x7f230c0f54b0 2024-01-11T05:04:48.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.586+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c045bb0 con 0x7f230c0f54b0 2024-01-11T05:04:48.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.586+0000 7f2301ffb700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f22f8031090 0x7f22f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:48.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.586+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2308041e00 con 0x7f230c0f54b0 2024-01-11T05:04:48.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.586+0000 7f2310e89700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f22f8031090 0x7f22f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:48.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.587+0000 7f2310e89700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f22f8031090 0x7f22f8033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f22fc0096a0 tx=0x7f22fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:48.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.591+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f230800f300 con 0x7f230c0f54b0 2024-01-11T05:04:48.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.797+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f230c107030 con 0x7f230c0f54b0 2024-01-11T05:04:48.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.798+0000 7f2301ffb700 1 -- 172.21.15.169:0/3222225813 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f230800f300 con 0x7f230c0f54b0 2024-01-11T05:04:48.798 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:48.798 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:48.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f22f8031090 msgr2=0x7f22f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:48.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f22f8031090 0x7f22f8033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f22fc0096a0 tx=0x7f22fc006b40).stop 2024-01-11T05:04:48.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 msgr2=0x7f230c1056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:48.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c1056a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2308000c00 tx=0x7f230800b040).stop 2024-01-11T05:04:48.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 shutdown_connections 2024-01-11T05:04:48.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f22f8031090 0x7f22f8033550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:48.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 --2- 172.21.15.169:0/3222225813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c0f54b0 0x7f230c1056a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:48.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 >> 172.21.15.169:0/3222225813 conn(0x7f230c0f0c10 msgr2=0x7f230c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:48.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 shutdown_connections 2024-01-11T05:04:48.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:48.799+0000 7f23138ee700 1 -- 172.21.15.169:0/3222225813 wait complete. 2024-01-11T05:04:48.801 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:48.280926+0000 mgr.smithi067.srbzpc (mgr.14156) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:48.798858+0000 mon.smithi067 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.169:0/3222225813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:50.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:50.171 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 -- 172.21.15.169:0/99678489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 msgr2=0x7efd200f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 --2- 172.21.15.169:0/99678489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f8fb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efd14009fb0 tx=0x7efd14009b30).stop 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 -- 172.21.15.169:0/99678489 shutdown_connections 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 --2- 172.21.15.169:0/99678489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f8fb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 -- 172.21.15.169:0/99678489 >> 172.21.15.169:0/99678489 conn(0x7efd200f0a30 msgr2=0x7efd200f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 -- 172.21.15.169:0/99678489 shutdown_connections 2024-01-11T05:04:51.432 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.432+0000 7efd1ffff700 1 -- 172.21.15.169:0/99678489 wait complete. 2024-01-11T05:04:51.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1ffff700 1 Processor -- start 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1ffff700 1 -- start start 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1ffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd200f4eb0 con 0x7efd20060360 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60590/0 (socket says 172.21.15.169:60590) 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1effd700 1 -- 172.21.15.169:0/2235860661 learned_addr learned my addr 172.21.15.169:0/2235860661 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1effd700 1 -- 172.21.15.169:0/2235860661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd14005040 con 0x7efd20060360 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.433+0000 7efd1effd700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f6a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7efd14009f80 tx=0x7efd1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd14009320 con 0x7efd20060360 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd200f5050 con 0x7efd20060360 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efd14013680 con 0x7efd20060360 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd1401eda0 con 0x7efd20060360 2024-01-11T05:04:51.435 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd200f54e0 con 0x7efd20060360 2024-01-11T05:04:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efd1401e420 con 0x7efd20060360 2024-01-11T05:04:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.434+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd200591e0 con 0x7efd20060360 2024-01-11T05:04:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.435+0000 7efd077fe700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efd080310e0 0x7efd080335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.435+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efd14042420 con 0x7efd20060360 2024-01-11T05:04:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.435+0000 7efd1e7fc700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efd080310e0 0x7efd080335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:51.436 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.435+0000 7efd1e7fc700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efd080310e0 0x7efd080335a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efd100096a0 tx=0x7efd10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:51.439 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.440+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efd1401d380 con 0x7efd20060360 2024-01-11T05:04:51.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.647+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd20045bb0 con 0x7efd20060360 2024-01-11T05:04:51.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.648+0000 7efd077fe700 1 -- 172.21.15.169:0/2235860661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd1401cb10 con 0x7efd20060360 2024-01-11T05:04:51.647 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:51.647 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:51.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efd080310e0 msgr2=0x7efd080335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:51.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efd080310e0 0x7efd080335a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efd100096a0 tx=0x7efd10006b40).stop 2024-01-11T05:04:51.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 msgr2=0x7efd200f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:51.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f6a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7efd14009f80 tx=0x7efd1400b040).stop 2024-01-11T05:04:51.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 shutdown_connections 2024-01-11T05:04:51.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efd080310e0 0x7efd080335a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:51.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 --2- 172.21.15.169:0/2235860661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20060360 0x7efd200f6a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:51.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 >> 172.21.15.169:0/2235860661 conn(0x7efd200f0a30 msgr2=0x7efd200f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:51.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 shutdown_connections 2024-01-11T05:04:51.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:51.649+0000 7efd1ffff700 1 -- 172.21.15.169:0/2235860661 wait complete. 2024-01-11T05:04:51.650 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:50.281215+0000 mgr.smithi067.srbzpc (mgr.14156) 46 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:52.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:51.647889+0000 mon.smithi067 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.169:0/2235860661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:53.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:53.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:53.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:52.281541+0000 mgr.smithi067.srbzpc (mgr.14156) 47 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 -- 172.21.15.169:0/209267311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 msgr2=0x7f278c0f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 --2- 172.21.15.169:0/209267311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c0f9200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2774009fb0 tx=0x7f2774009b30).stop 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 -- 172.21.15.169:0/209267311 shutdown_connections 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 --2- 172.21.15.169:0/209267311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c0f9200 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 -- 172.21.15.169:0/209267311 >> 172.21.15.169:0/209267311 conn(0x7f278c0f0be0 msgr2=0x7f278c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 -- 172.21.15.169:0/209267311 shutdown_connections 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.175+0000 7f2790d29700 1 -- 172.21.15.169:0/209267311 wait complete. 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f2790d29700 1 Processor -- start 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f2790d29700 1 -- start start 2024-01-11T05:04:54.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f2790d29700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:54.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f2790d29700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c108e00 con 0x7f278c0602c0 2024-01-11T05:04:54.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f278a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:54.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f278a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60618/0 (socket says 172.21.15.169:60618) 2024-01-11T05:04:54.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.176+0000 7f278a59c700 1 -- 172.21.15.169:0/423994660 learned_addr learned my addr 172.21.15.169:0/423994660 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:54.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f278a59c700 1 -- 172.21.15.169:0/423994660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2774005040 con 0x7f278c0602c0 2024-01-11T05:04:54.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f278a59c700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c108910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f2774013fd0 tx=0x7f2774009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:54.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2774018070 con 0x7f278c0602c0 2024-01-11T05:04:54.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c109000 con 0x7f278c0602c0 2024-01-11T05:04:54.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2774007b10 con 0x7f278c0602c0 2024-01-11T05:04:54.178 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f277401c3c0 con 0x7f278c0602c0 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.177+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c109490 con 0x7f278c0602c0 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.178+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2774007c80 con 0x7f278c0602c0 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.178+0000 7f2782ffd700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2778031090 0x7f2778033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.178+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c045bb0 con 0x7f278c0602c0 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.178+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2774042a10 con 0x7f278c0602c0 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.178+0000 7f2789d9b700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2778031090 0x7f2778033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:54.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.178+0000 7f2789d9b700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2778031090 0x7f2778033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f277c0096a0 tx=0x7f277c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:54.182 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.183+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2774025d20 con 0x7f278c0602c0 2024-01-11T05:04:54.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.394+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f278c10a310 con 0x7f278c0602c0 2024-01-11T05:04:54.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.394+0000 7f2782ffd700 1 -- 172.21.15.169:0/423994660 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f277401b370 con 0x7f278c0602c0 2024-01-11T05:04:54.394 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:54.394 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2778031090 msgr2=0x7f2778033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2778031090 0x7f2778033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f277c0096a0 tx=0x7f277c006b40).stop 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 msgr2=0x7f278c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c108910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f2774013fd0 tx=0x7f2774009b30).stop 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 shutdown_connections 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2778031090 0x7f2778033550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:54.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 --2- 172.21.15.169:0/423994660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278c0602c0 0x7f278c108910 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:54.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 >> 172.21.15.169:0/423994660 conn(0x7f278c0f0be0 msgr2=0x7f278c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:54.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 shutdown_connections 2024-01-11T05:04:54.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:54.396+0000 7f2790d29700 1 -- 172.21.15.169:0/423994660 wait complete. 2024-01-11T05:04:54.397 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:54.281913+0000 mgr.smithi067.srbzpc (mgr.14156) 48 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:54.394716+0000 mon.smithi067 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.169:0/423994660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:55.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:55.774 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:56.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 -- 172.21.15.169:0/1383299267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 msgr2=0x7f37f00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:56.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 --2- 172.21.15.169:0/1383299267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f00f69b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f37e0004760 tx=0x7f37e0009ad0).stop 2024-01-11T05:04:56.938 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 -- 172.21.15.169:0/1383299267 shutdown_connections 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 --2- 172.21.15.169:0/1383299267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f00f69b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 -- 172.21.15.169:0/1383299267 >> 172.21.15.169:0/1383299267 conn(0x7f37f00f1cd0 msgr2=0x7f37f00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 -- 172.21.15.169:0/1383299267 shutdown_connections 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.937+0000 7f37f7b35700 1 -- 172.21.15.169:0/1383299267 wait complete. 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f7b35700 1 Processor -- start 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f7b35700 1 -- start start 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f7b35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f7b35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f0069f00 con 0x7f37f00f6590 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f58d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f58d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60642/0 (socket says 172.21.15.169:60642) 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f58d1700 1 -- 172.21.15.169:0/1096975691 learned_addr learned my addr 172.21.15.169:0/1096975691 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f58d1700 1 -- 172.21.15.169:0/1096975691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e0009d70 con 0x7f37f00f6590 2024-01-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.938+0000 7f37f58d1700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f006ba80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f37e0004170 tx=0x7f37e0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.939+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e000e040 con 0x7f37f00f6590 2024-01-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.939+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f006a0a0 con 0x7f37f00f6590 2024-01-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.939+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f37e0008ef0 con 0x7f37f00f6590 2024-01-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.939+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e0014c90 con 0x7f37f00f6590 2024-01-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.939+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f006a530 con 0x7f37f00f6590 2024-01-11T05:04:56.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.940+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f37e0014420 con 0x7f37f00f6590 2024-01-11T05:04:56.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.940+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f00591e0 con 0x7f37f00f6590 2024-01-11T05:04:56.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.940+0000 7f37e67fc700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f37dc031090 0x7f37dc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:56.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.940+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f37e00421f0 con 0x7f37f00f6590 2024-01-11T05:04:56.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.940+0000 7f37f50d0700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f37dc031090 0x7f37dc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:56.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.941+0000 7f37f50d0700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f37dc031090 0x7f37dc033550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f37ec0096a0 tx=0x7f37ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:56.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:56.945+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f37e0026300 con 0x7f37f00f6590 2024-01-11T05:04:57.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.155+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37f00fc080 con 0x7f37f00f6590 2024-01-11T05:04:57.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.156+0000 7f37e67fc700 1 -- 172.21.15.169:0/1096975691 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f37e0012b20 con 0x7f37f00f6590 2024-01-11T05:04:57.156 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:57.156 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:57.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f37dc031090 msgr2=0x7f37dc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:57.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f37dc031090 0x7f37dc033550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f37ec0096a0 tx=0x7f37ec006b40).stop 2024-01-11T05:04:57.157 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 msgr2=0x7f37f006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f006ba80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f37e0004170 tx=0x7f37e0009ad0).stop 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 shutdown_connections 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f37dc031090 0x7f37dc033550 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 --2- 172.21.15.169:0/1096975691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f00f6590 0x7f37f006ba80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 >> 172.21.15.169:0/1096975691 conn(0x7f37f00f1cd0 msgr2=0x7f37f00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 shutdown_connections 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:57.158+0000 7f37f7b35700 1 -- 172.21.15.169:0/1096975691 wait complete. 2024-01-11T05:04:57.158 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:04:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:56.282314+0000 mgr.smithi067.srbzpc (mgr.14156) 49 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:57.156394+0000 mon.smithi067 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.169:0/1096975691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:04:58.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:04:58.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:04:59.751 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 -- 172.21.15.169:0/2903094885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 msgr2=0x7f10140f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 --2- 172.21.15.169:0/2903094885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f10140f69b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1010009fb0 tx=0x7f1010009b30).stop 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 -- 172.21.15.169:0/2903094885 shutdown_connections 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 --2- 172.21.15.169:0/2903094885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f10140f69b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 -- 172.21.15.169:0/2903094885 >> 172.21.15.169:0/2903094885 conn(0x7f10140f1cf0 msgr2=0x7f10140f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 -- 172.21.15.169:0/2903094885 shutdown_connections 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.749+0000 7f101b67f700 1 -- 172.21.15.169:0/2903094885 wait complete. 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101b67f700 1 Processor -- start 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101b67f700 1 -- start start 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101b67f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f1014106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:59.752 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101b67f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1014106c70 con 0x7f10140f6590 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101941b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f1014106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101941b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f1014106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43794/0 (socket says 172.21.15.169:43794) 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.750+0000 7f101941b700 1 -- 172.21.15.169:0/3026976995 learned_addr learned my addr 172.21.15.169:0/3026976995 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f101941b700 1 -- 172.21.15.169:0/3026976995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010005040 con 0x7f10140f6590 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f101941b700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f1014106780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f10100089f0 tx=0x7f101000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1010009320 con 0x7f10140f6590 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1014106e70 con 0x7f10140f6590 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1010013ec0 con 0x7f10140f6590 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1010014d30 con 0x7f10140f6590 2024-01-11T05:04:59.753 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.751+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1014107300 con 0x7f10140f6590 2024-01-11T05:04:59.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.752+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1010014420 con 0x7f10140f6590 2024-01-11T05:04:59.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.752+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1014045bb0 con 0x7f10140f6590 2024-01-11T05:04:59.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.752+0000 7f1009ffb700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1000031090 0x7f1000033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:04:59.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.752+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1010042400 con 0x7f10140f6590 2024-01-11T05:04:59.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.753+0000 7f1018c1a700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1000031090 0x7f1000033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:04:59.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.753+0000 7f1018c1a700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1000031090 0x7f1000033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f10040096a0 tx=0x7f1004006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:04:59.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.757+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f101001d980 con 0x7f10140f6590 2024-01-11T05:04:59.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:04:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:04:58.282772+0000 mgr.smithi067.srbzpc (mgr.14156) 50 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:04:59.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.962+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10141081b0 con 0x7f10140f6590 2024-01-11T05:04:59.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.963+0000 7f1009ffb700 1 -- 172.21.15.169:0/3026976995 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f101001d380 con 0x7f10140f6590 2024-01-11T05:04:59.964 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:04:59.964 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:04:59.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1000031090 msgr2=0x7f1000033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:59.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1000031090 0x7f1000033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f10040096a0 tx=0x7f1004006b40).stop 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 msgr2=0x7f1014106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f1014106780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f10100089f0 tx=0x7f101000b040).stop 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 shutdown_connections 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1000031090 0x7f1000033550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.964+0000 7f101b67f700 1 --2- 172.21.15.169:0/3026976995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140f6590 0x7f1014106780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.965+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 >> 172.21.15.169:0/3026976995 conn(0x7f10140f1cf0 msgr2=0x7f10140f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.965+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 shutdown_connections 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:04:59.965+0000 7f101b67f700 1 -- 172.21.15.169:0/3026976995 wait complete. 2024-01-11T05:04:59.966 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:00.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:00 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:04:59.964270+0000 mon.smithi067 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.169:0/3026976995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:01.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:01.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:00.283111+0000 mgr.smithi067.srbzpc (mgr.14156) 51 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:02.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 -- 172.21.15.169:0/2906491830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 msgr2=0x7efddc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:02.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 --2- 172.21.15.169:0/2906491830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc0f69b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7efdcc009fb0 tx=0x7efdcc009b30).stop 2024-01-11T05:05:02.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 -- 172.21.15.169:0/2906491830 shutdown_connections 2024-01-11T05:05:02.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 --2- 172.21.15.169:0/2906491830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc0f69b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:02.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 -- 172.21.15.169:0/2906491830 >> 172.21.15.169:0/2906491830 conn(0x7efddc0f1cf0 msgr2=0x7efddc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:02.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 -- 172.21.15.169:0/2906491830 shutdown_connections 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.457+0000 7efde14a0700 1 -- 172.21.15.169:0/2906491830 wait complete. 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.458+0000 7efde14a0700 1 Processor -- start 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.458+0000 7efde14a0700 1 -- start start 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.458+0000 7efde14a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.458+0000 7efde14a0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efddc106c70 con 0x7efddc0f6590 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efddaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efddaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43810/0 (socket says 172.21.15.169:43810) 2024-01-11T05:05:02.460 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efddaffd700 1 -- 172.21.15.169:0/280783081 learned_addr learned my addr 172.21.15.169:0/280783081 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efddaffd700 1 -- 172.21.15.169:0/280783081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc005040 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efddaffd700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc106780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7efdcc0089f0 tx=0x7efdcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc009320 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efddc106e70 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efdcc013ec0 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc014d30 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.459+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efddc107300 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.460+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efdcc014420 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.460+0000 7efdc37fe700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efdc4031090 0x7efdc4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.460+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efddc045bb0 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.460+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efdcc042400 con 0x7efddc0f6590 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.460+0000 7efdda7fc700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efdc4031090 0x7efdc4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:02.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.461+0000 7efdda7fc700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efdc4031090 0x7efdc4033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efdd00096a0 tx=0x7efdd0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:02.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.465+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efdcc01d980 con 0x7efddc0f6590 2024-01-11T05:05:02.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.675+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efddc108180 con 0x7efddc0f6590 2024-01-11T05:05:02.676 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.676+0000 7efdc37fe700 1 -- 172.21.15.169:0/280783081 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdcc02c090 con 0x7efddc0f6590 2024-01-11T05:05:02.677 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:02.677 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efdc4031090 msgr2=0x7efdc4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efdc4031090 0x7efdc4033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efdd00096a0 tx=0x7efdd0006b40).stop 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 msgr2=0x7efddc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc106780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7efdcc0089f0 tx=0x7efdcc00b040).stop 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 shutdown_connections 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efdc4031090 0x7efdc4033550 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 --2- 172.21.15.169:0/280783081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efddc0f6590 0x7efddc106780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 >> 172.21.15.169:0/280783081 conn(0x7efddc0f1cf0 msgr2=0x7efddc0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.677+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 shutdown_connections 2024-01-11T05:05:02.679 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:02.678+0000 7efde14a0700 1 -- 172.21.15.169:0/280783081 wait complete. 2024-01-11T05:05:02.680 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:02.283359+0000 mgr.smithi067.srbzpc (mgr.14156) 52 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:02.677165+0000 mon.smithi067 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.169:0/280783081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:04.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:04.052 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 -- 172.21.15.169:0/3865797413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 msgr2=0x7ff5e00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 --2- 172.21.15.169:0/3865797413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e00f69b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff5d0004760 tx=0x7ff5d0009ad0).stop 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 -- 172.21.15.169:0/3865797413 shutdown_connections 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 --2- 172.21.15.169:0/3865797413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e00f69b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 -- 172.21.15.169:0/3865797413 >> 172.21.15.169:0/3865797413 conn(0x7ff5e00f1cd0 msgr2=0x7ff5e00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 -- 172.21.15.169:0/3865797413 shutdown_connections 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.200+0000 7ff5e8643700 1 -- 172.21.15.169:0/3865797413 wait complete. 2024-01-11T05:05:05.202 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e8643700 1 Processor -- start 2024-01-11T05:05:05.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e8643700 1 -- start start 2024-01-11T05:05:05.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e8643700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:05.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e8643700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0069f00 con 0x7ff5e00f6590 2024-01-11T05:05:05.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e63df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:05.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e63df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43820/0 (socket says 172.21.15.169:43820) 2024-01-11T05:05:05.203 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.201+0000 7ff5e63df700 1 -- 172.21.15.169:0/1601482401 learned_addr learned my addr 172.21.15.169:0/1601482401 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:05.204 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5e63df700 1 -- 172.21.15.169:0/1601482401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0009d70 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5e63df700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e006ba80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff5d0004170 tx=0x7ff5d0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d000e040 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e006a0a0 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e006a530 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff5d0008ef0 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.202+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0014b20 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.203+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff5d0014c80 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.203+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c8004fa0 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.203+0000 7ff5d6ffd700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5cc031090 0x7ff5cc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.203+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff5d00421c0 con 0x7ff5e00f6590 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.203+0000 7ff5e5bde700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5cc031090 0x7ff5cc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:05.205 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.204+0000 7ff5e5bde700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5cc031090 0x7ff5cc033550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff5dc0096a0 tx=0x7ff5dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:05.209 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.208+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff5d0012b90 con 0x7ff5e00f6590 2024-01-11T05:05:05.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.416+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5c8005670 con 0x7ff5e00f6590 2024-01-11T05:05:05.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.416+0000 7ff5d6ffd700 1 -- 172.21.15.169:0/1601482401 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5d00127a0 con 0x7ff5e00f6590 2024-01-11T05:05:05.417 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:05.418 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5cc031090 msgr2=0x7ff5cc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5cc031090 0x7ff5cc033550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff5dc0096a0 tx=0x7ff5dc006b40).stop 2024-01-11T05:05:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 msgr2=0x7ff5e006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e006ba80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff5d0004170 tx=0x7ff5d0009ad0).stop 2024-01-11T05:05:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 shutdown_connections 2024-01-11T05:05:05.419 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5cc031090 0x7ff5cc033550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:05.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 --2- 172.21.15.169:0/1601482401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5e00f6590 0x7ff5e006ba80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:05.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 >> 172.21.15.169:0/1601482401 conn(0x7ff5e00f1cd0 msgr2=0x7ff5e00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:05.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 shutdown_connections 2024-01-11T05:05:05.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:05.418+0000 7ff5e8643700 1 -- 172.21.15.169:0/1601482401 wait complete. 2024-01-11T05:05:05.420 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:04.283692+0000 mgr.smithi067.srbzpc (mgr.14156) 53 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:06.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:06 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:05.417640+0000 mon.smithi067 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.169:0/1601482401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:06.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:06.892 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:07.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:06.284041+0000 mgr.smithi067.srbzpc (mgr.14156) 54 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 -- 172.21.15.169:0/3298319552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 msgr2=0x7ff9d00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 --2- 172.21.15.169:0/3298319552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d00f58a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff9c0009fb0 tx=0x7ff9c0009b30).stop 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 -- 172.21.15.169:0/3298319552 shutdown_connections 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 --2- 172.21.15.169:0/3298319552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d00f58a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 -- 172.21.15.169:0/3298319552 >> 172.21.15.169:0/3298319552 conn(0x7ff9d00f0be0 msgr2=0x7ff9d00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 -- 172.21.15.169:0/3298319552 shutdown_connections 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.058+0000 7ff9d5169700 1 -- 172.21.15.169:0/3298319552 wait complete. 2024-01-11T05:05:08.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.059+0000 7ff9d5169700 1 Processor -- start 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.059+0000 7ff9d5169700 1 -- start start 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.059+0000 7ff9d5169700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.059+0000 7ff9d5169700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0060300 con 0x7ff9d00f5480 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.059+0000 7ff9ced9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9ced9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43836/0 (socket says 172.21.15.169:43836) 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9ced9d700 1 -- 172.21.15.169:0/3225306546 learned_addr learned my addr 172.21.15.169:0/3225306546 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9ced9d700 1 -- 172.21.15.169:0/3225306546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0005040 con 0x7ff9d00f5480 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9ced9d700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d0109510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff9c00089f0 tx=0x7ff9c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c0009320 con 0x7ff9d00f5480 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d0060500 con 0x7ff9d00f5480 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff9c0013ec0 con 0x7ff9d00f5480 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c0014d30 con 0x7ff9d00f5480 2024-01-11T05:05:08.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.060+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d0109bc0 con 0x7ff9d00f5480 2024-01-11T05:05:08.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.061+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff9c00095f0 con 0x7ff9d00f5480 2024-01-11T05:05:08.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.061+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d00591e0 con 0x7ff9d00f5480 2024-01-11T05:05:08.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.061+0000 7ff9c77fe700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff9bc031090 0x7ff9bc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:08.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.061+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff9c00424e0 con 0x7ff9d00f5480 2024-01-11T05:05:08.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.062+0000 7ff9ce59c700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff9bc031090 0x7ff9bc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:08.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.062+0000 7ff9ce59c700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff9bc031090 0x7ff9bc033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff9b80096a0 tx=0x7ff9b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:08.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.066+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff9c001d7f0 con 0x7ff9d00f5480 2024-01-11T05:05:08.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.275+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9d0045bb0 con 0x7ff9d00f5480 2024-01-11T05:05:08.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.275+0000 7ff9c77fe700 1 -- 172.21.15.169:0/3225306546 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9c000fe40 con 0x7ff9d00f5480 2024-01-11T05:05:08.276 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:08.276 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff9bc031090 msgr2=0x7ff9bc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff9bc031090 0x7ff9bc033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff9b80096a0 tx=0x7ff9b8006b40).stop 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 msgr2=0x7ff9d0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d0109510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff9c00089f0 tx=0x7ff9c000b040).stop 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 shutdown_connections 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff9bc031090 0x7ff9bc033550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:08.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 --2- 172.21.15.169:0/3225306546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9d00f5480 0x7ff9d0109510 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:08.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 >> 172.21.15.169:0/3225306546 conn(0x7ff9d00f0be0 msgr2=0x7ff9d00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:08.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 shutdown_connections 2024-01-11T05:05:08.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:08.277+0000 7ff9d5169700 1 -- 172.21.15.169:0/3225306546 wait complete. 2024-01-11T05:05:08.279 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:08.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:08.276723+0000 mon.smithi067 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.169:0/3225306546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:09.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:09.658 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:08.284424+0000 mgr.smithi067.srbzpc (mgr.14156) 55 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:10.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 -- 172.21.15.169:0/1099908567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 msgr2=0x7f26980f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:10.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 --2- 172.21.15.169:0/1099908567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f26980f56f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2688009fb0 tx=0x7f2688009b30).stop 2024-01-11T05:05:10.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 -- 172.21.15.169:0/1099908567 shutdown_connections 2024-01-11T05:05:10.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 --2- 172.21.15.169:0/1099908567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f26980f56f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:10.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 -- 172.21.15.169:0/1099908567 >> 172.21.15.169:0/1099908567 conn(0x7f26980f0a30 msgr2=0x7f26980f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:10.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 -- 172.21.15.169:0/1099908567 shutdown_connections 2024-01-11T05:05:10.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 -- 172.21.15.169:0/1099908567 wait complete. 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 Processor -- start 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.771+0000 7f269d663700 1 -- start start 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f269d663700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f2698106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f269d663700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698106af0 con 0x7f26980f52d0 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2697fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f2698106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2697fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f2698106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:44676/0 (socket says 172.21.15.169:44676) 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2697fff700 1 -- 172.21.15.169:0/1688197521 learned_addr learned my addr 172.21.15.169:0/1688197521 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:10.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2697fff700 1 -- 172.21.15.169:0/1688197521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688005040 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2697fff700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f2698106600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f26880089f0 tx=0x7f268800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2688009320 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698106cf0 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2688013ec0 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2688014d30 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.772+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2698107180 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.773+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f26880095f0 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.773+0000 7f2694ff9700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2684031090 0x7f2684033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.773+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2688042500 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.773+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2698045bb0 con 0x7f26980f52d0 2024-01-11T05:05:10.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.774+0000 7f26977fe700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2684031090 0x7f2684033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:10.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.774+0000 7f26977fe700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2684031090 0x7f2684033550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f26800096a0 tx=0x7f2680006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:10.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.778+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f268801dbe0 con 0x7f26980f52d0 2024-01-11T05:05:10.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.985+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26980faeb0 con 0x7f26980f52d0 2024-01-11T05:05:10.987 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.986+0000 7f2694ff9700 1 -- 172.21.15.169:0/1688197521 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f268802b020 con 0x7f26980f52d0 2024-01-11T05:05:10.987 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:10.987 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:10.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2684031090 msgr2=0x7f2684033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:10.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2684031090 0x7f2684033550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f26800096a0 tx=0x7f2680006b40).stop 2024-01-11T05:05:10.988 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 msgr2=0x7f2698106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f2698106600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f26880089f0 tx=0x7f268800b040).stop 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 shutdown_connections 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2684031090 0x7f2684033550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 --2- 172.21.15.169:0/1688197521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26980f52d0 0x7f2698106600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 >> 172.21.15.169:0/1688197521 conn(0x7f26980f0a30 msgr2=0x7f269805fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 shutdown_connections 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:10.988+0000 7f269d663700 1 -- 172.21.15.169:0/1688197521 wait complete. 2024-01-11T05:05:10.989 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:10.284862+0000 mgr.smithi067.srbzpc (mgr.14156) 56 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:10.987428+0000 mon.smithi067 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.169:0/1688197521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:12.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:12.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.537+0000 7fc118f2e700 1 -- 172.21.15.169:0/740681396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 msgr2=0x7fc1140f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.537+0000 7fc118f2e700 1 --2- 172.21.15.169:0/740681396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc1140f6800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fc104009fb0 tx=0x7fc104009b30).stop 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.537+0000 7fc118f2e700 1 -- 172.21.15.169:0/740681396 shutdown_connections 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.537+0000 7fc118f2e700 1 --2- 172.21.15.169:0/740681396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc1140f6800 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.537+0000 7fc118f2e700 1 -- 172.21.15.169:0/740681396 >> 172.21.15.169:0/740681396 conn(0x7fc1140f1b40 msgr2=0x7fc1140f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.538+0000 7fc118f2e700 1 -- 172.21.15.169:0/740681396 shutdown_connections 2024-01-11T05:05:13.539 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.538+0000 7fc118f2e700 1 -- 172.21.15.169:0/740681396 wait complete. 2024-01-11T05:05:13.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.538+0000 7fc118f2e700 1 Processor -- start 2024-01-11T05:05:13.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.538+0000 7fc118f2e700 1 -- start start 2024-01-11T05:05:13.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.538+0000 7fc118f2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc114106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:13.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.538+0000 7fc118f2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc114106b10 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.539+0000 7fc1137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc114106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.539+0000 7fc1137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc114106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:44702/0 (socket says 172.21.15.169:44702) 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.539+0000 7fc1137fe700 1 -- 172.21.15.169:0/767159223 learned_addr learned my addr 172.21.15.169:0/767159223 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.539+0000 7fc1137fe700 1 -- 172.21.15.169:0/767159223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc104005040 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.539+0000 7fc1137fe700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc114106620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc1040089f0 tx=0x7fc10400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.539+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc104009320 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.540+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc104013ec0 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.540+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc114106d10 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.540+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc104014d30 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.540+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1141071a0 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.540+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc1040095f0 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.540+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc114045bb0 con 0x7fc1140f63e0 2024-01-11T05:05:13.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.541+0000 7fc0fbfff700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc0fc0310e0 0x7fc0fc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:13.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.541+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc1040424a0 con 0x7fc1140f63e0 2024-01-11T05:05:13.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.541+0000 7fc112ffd700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc0fc0310e0 0x7fc0fc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:13.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.541+0000 7fc112ffd700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc0fc0310e0 0x7fc0fc0335a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc1080096a0 tx=0x7fc108006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:13.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.546+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc10401d7f0 con 0x7fc1140f63e0 2024-01-11T05:05:13.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.753+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc114108050 con 0x7fc1140f63e0 2024-01-11T05:05:13.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.754+0000 7fc0fbfff700 1 -- 172.21.15.169:0/767159223 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc10401d7f0 con 0x7fc1140f63e0 2024-01-11T05:05:13.755 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:13.755 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:13.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc0fc0310e0 msgr2=0x7fc0fc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:13.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc0fc0310e0 0x7fc0fc0335a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc1080096a0 tx=0x7fc108006b40).stop 2024-01-11T05:05:13.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 msgr2=0x7fc114106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:13.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc114106620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc1040089f0 tx=0x7fc10400b040).stop 2024-01-11T05:05:13.756 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 shutdown_connections 2024-01-11T05:05:13.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc0fc0310e0 0x7fc0fc0335a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:13.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 --2- 172.21.15.169:0/767159223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1140f63e0 0x7fc114106620 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:13.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 >> 172.21.15.169:0/767159223 conn(0x7fc1140f1b40 msgr2=0x7fc1140f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:13.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 shutdown_connections 2024-01-11T05:05:13.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:13.755+0000 7fc118f2e700 1 -- 172.21.15.169:0/767159223 wait complete. 2024-01-11T05:05:13.757 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:12.285272+0000 mgr.smithi067.srbzpc (mgr.14156) 57 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:14.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:14 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:13.755141+0000 mon.smithi067 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.169:0/767159223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:15.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:15.184 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:15.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:14.285572+0000 mgr.smithi067.srbzpc (mgr.14156) 58 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:16.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 -- 172.21.15.169:0/4289846129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 msgr2=0x7f93940f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:16.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 --2- 172.21.15.169:0/4289846129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f93940f6800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9388009fb0 tx=0x7f9388009b30).stop 2024-01-11T05:05:16.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 -- 172.21.15.169:0/4289846129 shutdown_connections 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 --2- 172.21.15.169:0/4289846129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f93940f6800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 -- 172.21.15.169:0/4289846129 >> 172.21.15.169:0/4289846129 conn(0x7f93940f1b40 msgr2=0x7f93940f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 -- 172.21.15.169:0/4289846129 shutdown_connections 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.380+0000 7f939885b700 1 -- 172.21.15.169:0/4289846129 wait complete. 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.381+0000 7f939885b700 1 Processor -- start 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.381+0000 7f939885b700 1 -- start start 2024-01-11T05:05:16.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f939885b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f9394106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f939885b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394106b10 con 0x7f93940f63e0 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f9392d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f9394106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f9392d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f9394106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:44720/0 (socket says 172.21.15.169:44720) 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f9392d9d700 1 -- 172.21.15.169:0/2584163944 learned_addr learned my addr 172.21.15.169:0/2584163944 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f9392d9d700 1 -- 172.21.15.169:0/2584163944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388005040 con 0x7f93940f63e0 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.382+0000 7f9392d9d700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f9394106620 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9388000c00 tx=0x7f938800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.383+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93880095f0 con 0x7f93940f63e0 2024-01-11T05:05:16.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.383+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9394106d10 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.383+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9388007e50 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.383+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9388014e70 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.383+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93941071a0 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.384+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9388025070 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.384+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9394045bb0 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.384+0000 7f937b7fe700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f937c031090 0x7f937c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.384+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9388041d60 con 0x7f93940f63e0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.384+0000 7f939259c700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f937c031090 0x7f937c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:16.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.384+0000 7f939259c700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f937c031090 0x7f937c033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f93840096a0 tx=0x7f9384006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:16.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.388+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f938800f4e0 con 0x7f93940f63e0 2024-01-11T05:05:16.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.595+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9394107fa0 con 0x7f93940f63e0 2024-01-11T05:05:16.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.596+0000 7f937b7fe700 1 -- 172.21.15.169:0/2584163944 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f938800f300 con 0x7f93940f63e0 2024-01-11T05:05:16.597 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:16.597 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f937c031090 msgr2=0x7f937c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f937c031090 0x7f937c033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f93840096a0 tx=0x7f9384006b40).stop 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 msgr2=0x7f9394106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f9394106620 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9388000c00 tx=0x7f938800b040).stop 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 shutdown_connections 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f937c031090 0x7f937c033550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 --2- 172.21.15.169:0/2584163944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93940f63e0 0x7f9394106620 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 >> 172.21.15.169:0/2584163944 conn(0x7f93940f1b40 msgr2=0x7f93940f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 shutdown_connections 2024-01-11T05:05:16.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:16.598+0000 7f939885b700 1 -- 172.21.15.169:0/2584163944 wait complete. 2024-01-11T05:05:16.600 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:17.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:16.285842+0000 mgr.smithi067.srbzpc (mgr.14156) 59 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:17.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:16.597444+0000 mon.smithi067 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.169:0/2584163944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:17.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:17.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:19.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 -- 172.21.15.169:0/2028596656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 msgr2=0x7f9f8c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:19.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 --2- 172.21.15.169:0/2028596656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c0f69b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9f7c009a00 tx=0x7f9f7c004cb0).stop 2024-01-11T05:05:19.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 -- 172.21.15.169:0/2028596656 shutdown_connections 2024-01-11T05:05:19.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 --2- 172.21.15.169:0/2028596656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c0f69b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:19.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 -- 172.21.15.169:0/2028596656 >> 172.21.15.169:0/2028596656 conn(0x7f9f8c0f1cd0 msgr2=0x7f9f8c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:19.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 -- 172.21.15.169:0/2028596656 shutdown_connections 2024-01-11T05:05:19.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.128+0000 7f9f94766700 1 -- 172.21.15.169:0/2028596656 wait complete. 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f94766700 1 Processor -- start 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f94766700 1 -- start start 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f94766700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f94766700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c106c60 con 0x7f9f8c0f6590 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f92502700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f92502700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:44752/0 (socket says 172.21.15.169:44752) 2024-01-11T05:05:19.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.129+0000 7f9f92502700 1 -- 172.21.15.169:0/654110134 learned_addr learned my addr 172.21.15.169:0/654110134 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f92502700 1 -- 172.21.15.169:0/654110134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c009640 con 0x7f9f8c0f6590 2024-01-11T05:05:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f92502700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c106770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9f7c009610 tx=0x7f9f7c004cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7c0092e0 con 0x7f9f8c0f6590 2024-01-11T05:05:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c106e60 con 0x7f9f8c0f6590 2024-01-11T05:05:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9f7c007d50 con 0x7f9f8c0f6590 2024-01-11T05:05:19.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7c003de0 con 0x7f9f8c0f6590 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.130+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c1072f0 con 0x7f9f8c0f6590 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.131+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9f7c003650 con 0x7f9f8c0f6590 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.131+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c045bb0 con 0x7f9f8c0f6590 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.131+0000 7f9f8affd700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f74031090 0x7f9f74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.131+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9f7c042c60 con 0x7f9f8c0f6590 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.131+0000 7f9f91d01700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f74031090 0x7f9f74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:19.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.132+0000 7f9f91d01700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f74031090 0x7f9f74033550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f9f800096a0 tx=0x7f9f80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:19.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.136+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9f7c010960 con 0x7f9f8c0f6590 2024-01-11T05:05:19.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.342+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f8c1081a0 con 0x7f9f8c0f6590 2024-01-11T05:05:19.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.343+0000 7f9f8affd700 1 -- 172.21.15.169:0/654110134 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f7c010310 con 0x7f9f8c0f6590 2024-01-11T05:05:19.344 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:19.344 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f74031090 msgr2=0x7f9f74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f74031090 0x7f9f74033550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f9f800096a0 tx=0x7f9f80006b40).stop 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 msgr2=0x7f9f8c106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c106770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9f7c009610 tx=0x7f9f7c004cb0).stop 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 shutdown_connections 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f74031090 0x7f9f74033550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 --2- 172.21.15.169:0/654110134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c0f6590 0x7f9f8c106770 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:19.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 >> 172.21.15.169:0/654110134 conn(0x7f9f8c0f1cd0 msgr2=0x7f9f8c0f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:19.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 shutdown_connections 2024-01-11T05:05:19.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:19.344+0000 7f9f94766700 1 -- 172.21.15.169:0/654110134 wait complete. 2024-01-11T05:05:19.346 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:18.286214+0000 mgr.smithi067.srbzpc (mgr.14156) 60 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:20.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:20.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:20.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:20 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:19.344073+0000 mon.smithi067 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.169:0/654110134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:20.286505+0000 mgr.smithi067.srbzpc (mgr.14156) 61 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:21.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 -- 172.21.15.169:0/1608090672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 msgr2=0x7f14140f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 --2- 172.21.15.169:0/1608090672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f14140f9160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f13fc009fb0 tx=0x7f13fc009b30).stop 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 -- 172.21.15.169:0/1608090672 shutdown_connections 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 --2- 172.21.15.169:0/1608090672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f14140f9160 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 -- 172.21.15.169:0/1608090672 >> 172.21.15.169:0/1608090672 conn(0x7f14140f0be0 msgr2=0x7f14140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 -- 172.21.15.169:0/1608090672 shutdown_connections 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.903+0000 7f1419ae6700 1 -- 172.21.15.169:0/1608090672 wait complete. 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f1419ae6700 1 Processor -- start 2024-01-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f1419ae6700 1 -- start start 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f1419ae6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f1414106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f1419ae6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1414106c40 con 0x7f1414060360 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f14137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f1414106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f14137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f1414106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43344/0 (socket says 172.21.15.169:43344) 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.904+0000 7f14137fe700 1 -- 172.21.15.169:0/3225620029 learned_addr learned my addr 172.21.15.169:0/3225620029 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f14137fe700 1 -- 172.21.15.169:0/3225620029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc005040 con 0x7f1414060360 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f14137fe700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f1414106750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f13fc009f80 tx=0x7f13fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:21.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13fc009320 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1414106e40 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f13fc013680 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13fc01eda0 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.905+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14141072d0 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.906+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f13fc014420 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.906+0000 7f140bfff700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1400031090 0x7f1400033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.906+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13fc042420 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.906+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14140591e0 con 0x7f1414060360 2024-01-11T05:05:21.908 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.906+0000 7f1412ffd700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1400031090 0x7f1400033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:21.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.907+0000 7f1412ffd700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1400031090 0x7f1400033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f14040096a0 tx=0x7f1404006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:21.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:21.911+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f13fc014700 con 0x7f1414060360 2024-01-11T05:05:22.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.117+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1414045bb0 con 0x7f1414060360 2024-01-11T05:05:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.117+0000 7f140bfff700 1 -- 172.21.15.169:0/3225620029 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13fc00fb30 con 0x7f1414060360 2024-01-11T05:05:22.118 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:22.118 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1400031090 msgr2=0x7f1400033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1400031090 0x7f1400033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f14040096a0 tx=0x7f1404006b40).stop 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 msgr2=0x7f1414106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f1414106750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f13fc009f80 tx=0x7f13fc00b040).stop 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 shutdown_connections 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1400031090 0x7f1400033550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 --2- 172.21.15.169:0/3225620029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1414060360 0x7f1414106750 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 >> 172.21.15.169:0/3225620029 conn(0x7f14140f0be0 msgr2=0x7f14140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 shutdown_connections 2024-01-11T05:05:22.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:22.119+0000 7f1419ae6700 1 -- 172.21.15.169:0/3225620029 wait complete. 2024-01-11T05:05:22.121 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:22.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:22 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:22.118659+0000 mon.smithi067 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.169:0/3225620029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:23.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:23.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:23.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:22.286772+0000 mgr.smithi067.srbzpc (mgr.14156) 62 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:23.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:22.412932+0000 mon.smithi067 (mon.0) 145 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:05:23.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:22.413787+0000 mon.smithi067 (mon.0) 146 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:05:24.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 -- 172.21.15.169:0/2867575997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 msgr2=0x7f26340f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 --2- 172.21.15.169:0/2867575997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26340f58a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f2624009960 tx=0x7f2624004d10).stop 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 -- 172.21.15.169:0/2867575997 shutdown_connections 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 --2- 172.21.15.169:0/2867575997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26340f58a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 -- 172.21.15.169:0/2867575997 >> 172.21.15.169:0/2867575997 conn(0x7f26340f0be0 msgr2=0x7f26340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 -- 172.21.15.169:0/2867575997 shutdown_connections 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.682+0000 7f2639c9b700 1 -- 172.21.15.169:0/2867575997 wait complete. 2024-01-11T05:05:24.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.683+0000 7f2639c9b700 1 Processor -- start 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.683+0000 7f2639c9b700 1 -- start start 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.683+0000 7f2639c9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26341094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.683+0000 7f2639c9b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634060320 con 0x7f26340f5480 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f26337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26341094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f26337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26341094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43350/0 (socket says 172.21.15.169:43350) 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f26337fe700 1 -- 172.21.15.169:0/3860850910 learned_addr learned my addr 172.21.15.169:0/3860850910 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f26337fe700 1 -- 172.21.15.169:0/3860850910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624009640 con 0x7f26340f5480 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f26337fe700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26341094f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f2624009610 tx=0x7f2624004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:24.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2624018070 con 0x7f26340f5480 2024-01-11T05:05:24.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2634060520 con 0x7f26340f5480 2024-01-11T05:05:24.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f262400b7e0 con 0x7f26340f5480 2024-01-11T05:05:24.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f262401d420 con 0x7f26340f5480 2024-01-11T05:05:24.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.684+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2634109ba0 con 0x7f26340f5480 2024-01-11T05:05:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.685+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2624024060 con 0x7f26340f5480 2024-01-11T05:05:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.685+0000 7f262bfff700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2620031090 0x7f2620033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.685+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2634045bb0 con 0x7f26340f5480 2024-01-11T05:05:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.686+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f26240425f0 con 0x7f26340f5480 2024-01-11T05:05:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.686+0000 7f2632ffd700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2620031090 0x7f2620033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:24.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.686+0000 7f2632ffd700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2620031090 0x7f2620033550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:24.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.690+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2624026c50 con 0x7f26340f5480 2024-01-11T05:05:24.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.896+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26340fb060 con 0x7f26340f5480 2024-01-11T05:05:24.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.897+0000 7f262bfff700 1 -- 172.21.15.169:0/3860850910 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2624024e20 con 0x7f26340f5480 2024-01-11T05:05:24.898 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:24.898 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:24.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2620031090 msgr2=0x7f2620033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2620031090 0x7f2620033550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).stop 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 msgr2=0x7f26341094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26341094f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f2624009610 tx=0x7f2624004d10).stop 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 shutdown_connections 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2620031090 0x7f2620033550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 --2- 172.21.15.169:0/3860850910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26340f5480 0x7f26341094f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 >> 172.21.15.169:0/3860850910 conn(0x7f26340f0be0 msgr2=0x7f26340f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 shutdown_connections 2024-01-11T05:05:24.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:24.899+0000 7f2639c9b700 1 -- 172.21.15.169:0/3860850910 wait complete. 2024-01-11T05:05:24.901 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:24.287174+0000 mgr.smithi067.srbzpc (mgr.14156) 63 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:24.898034+0000 mon.smithi067 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.169:0/3860850910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:26.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:26.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:27.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 -- 172.21.15.169:0/1093490822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 msgr2=0x7fb6800f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:27.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 --2- 172.21.15.169:0/1093490822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb6800f67f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb67c009fb0 tx=0x7fb67c009b30).stop 2024-01-11T05:05:27.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 -- 172.21.15.169:0/1093490822 shutdown_connections 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 --2- 172.21.15.169:0/1093490822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb6800f67f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 -- 172.21.15.169:0/1093490822 >> 172.21.15.169:0/1093490822 conn(0x7fb6800f1b10 msgr2=0x7fb6800f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 -- 172.21.15.169:0/1093490822 shutdown_connections 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 -- 172.21.15.169:0/1093490822 wait complete. 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 Processor -- start 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 -- start start 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.381+0000 7fb687134700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb680106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb687134700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb680106b30 con 0x7fb6800f63d0 2024-01-11T05:05:27.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb686132700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb680106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb686132700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb680106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43368/0 (socket says 172.21.15.169:43368) 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb686132700 1 -- 172.21.15.169:0/159287092 learned_addr learned my addr 172.21.15.169:0/159287092 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb686132700 1 -- 172.21.15.169:0/159287092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c005040 con 0x7fb6800f63d0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb686132700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb680106640 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb67c000c00 tx=0x7fb67c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c0095f0 con 0x7fb6800f63d0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb680106d30 con 0x7fb6800f63d0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb67c007e50 con 0x7fb6800f63d0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c014e70 con 0x7fb6800f63d0 2024-01-11T05:05:27.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.382+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6801071c0 con 0x7fb6800f63d0 2024-01-11T05:05:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.383+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb67c0079c0 con 0x7fb6800f63d0 2024-01-11T05:05:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.383+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb680045bb0 con 0x7fb6800f63d0 2024-01-11T05:05:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.383+0000 7fb676ffd700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb66c031090 0x7fb66c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.383+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb67c041c50 con 0x7fb6800f63d0 2024-01-11T05:05:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.384+0000 7fb685931700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb66c031090 0x7fb66c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:27.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.384+0000 7fb685931700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb66c031090 0x7fb66c033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb6700096a0 tx=0x7fb670006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.388+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb67c01b910 con 0x7fb6800f63d0 2024-01-11T05:05:27.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.598+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb680108030 con 0x7fb6800f63d0 2024-01-11T05:05:27.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.598+0000 7fb676ffd700 1 -- 172.21.15.169:0/159287092 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb67c01b2c0 con 0x7fb6800f63d0 2024-01-11T05:05:27.599 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:27.599 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb66c031090 msgr2=0x7fb66c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb66c031090 0x7fb66c033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb6700096a0 tx=0x7fb670006b40).stop 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 msgr2=0x7fb680106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb680106640 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb67c000c00 tx=0x7fb67c00b040).stop 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 shutdown_connections 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb66c031090 0x7fb66c033550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 --2- 172.21.15.169:0/159287092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6800f63d0 0x7fb680106640 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:27.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 >> 172.21.15.169:0/159287092 conn(0x7fb6800f1b10 msgr2=0x7fb6800f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 shutdown_connections 2024-01-11T05:05:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:27.600+0000 7fb687134700 1 -- 172.21.15.169:0/159287092 wait complete. 2024-01-11T05:05:27.602 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:26.287502+0000 mgr.smithi067.srbzpc (mgr.14156) 64 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:28.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:28 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:27.599823+0000 mon.smithi067 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.169:0/159287092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:28.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:28.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:28.287785+0000 mgr.smithi067.srbzpc (mgr.14156) 65 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:30.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 -- 172.21.15.169:0/868099042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 msgr2=0x7ff4bc0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:30.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 --2- 172.21.15.169:0/868099042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc0f9160 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff4ac009fb0 tx=0x7ff4ac009b30).stop 2024-01-11T05:05:30.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 -- 172.21.15.169:0/868099042 shutdown_connections 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 --2- 172.21.15.169:0/868099042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc0f9160 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 -- 172.21.15.169:0/868099042 >> 172.21.15.169:0/868099042 conn(0x7ff4bc0f0be0 msgr2=0x7ff4bc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 -- 172.21.15.169:0/868099042 shutdown_connections 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 -- 172.21.15.169:0/868099042 wait complete. 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.076+0000 7ff4c400c700 1 Processor -- start 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c400c700 1 -- start start 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c400c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc108960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c400c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc108e50 con 0x7ff4bc060360 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c1da8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c1da8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40074/0 (socket says 172.21.15.169:40074) 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c1da8700 1 -- 172.21.15.169:0/1705054238 learned_addr learned my addr 172.21.15.169:0/1705054238 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c1da8700 1 -- 172.21.15.169:0/1705054238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac005040 con 0x7ff4bc060360 2024-01-11T05:05:30.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.077+0000 7ff4c1da8700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc108960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff4ac009f80 tx=0x7ff4ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:30.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.078+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4ac009320 con 0x7ff4bc060360 2024-01-11T05:05:30.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.078+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc109050 con 0x7ff4bc060360 2024-01-11T05:05:30.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.078+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff4ac013680 con 0x7ff4bc060360 2024-01-11T05:05:30.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.078+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4ac01eda0 con 0x7ff4bc060360 2024-01-11T05:05:30.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.078+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc1094e0 con 0x7ff4bc060360 2024-01-11T05:05:30.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.078+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff4ac01e420 con 0x7ff4bc060360 2024-01-11T05:05:30.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.079+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc045bb0 con 0x7ff4bc060360 2024-01-11T05:05:30.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.079+0000 7ff4b27fc700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff4a8031090 0x7ff4a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:30.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.079+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff4ac042420 con 0x7ff4bc060360 2024-01-11T05:05:30.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.079+0000 7ff4c15a7700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff4a8031090 0x7ff4a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:30.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.079+0000 7ff4c15a7700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff4a8031090 0x7ff4a8033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff4b80096a0 tx=0x7ff4b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.084+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff4ac01d380 con 0x7ff4bc060360 2024-01-11T05:05:30.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.293+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4bc10a330 con 0x7ff4bc060360 2024-01-11T05:05:30.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4b27fc700 1 -- 172.21.15.169:0/1705054238 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4ac01cb10 con 0x7ff4bc060360 2024-01-11T05:05:30.295 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:30.295 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff4a8031090 msgr2=0x7ff4a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff4a8031090 0x7ff4a8033550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff4b80096a0 tx=0x7ff4b8006b40).stop 2024-01-11T05:05:30.296 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 msgr2=0x7ff4bc108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc108960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff4ac009f80 tx=0x7ff4ac00b040).stop 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 shutdown_connections 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff4a8031090 0x7ff4a8033550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 --2- 172.21.15.169:0/1705054238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4bc060360 0x7ff4bc108960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.295+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 >> 172.21.15.169:0/1705054238 conn(0x7ff4bc0f0be0 msgr2=0x7ff4bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.296+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 shutdown_connections 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:30.296+0000 7ff4c400c700 1 -- 172.21.15.169:0/1705054238 wait complete. 2024-01-11T05:05:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:30.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:30 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:30.295138+0000 mon.smithi067 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.169:0/1705054238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:31.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:31.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:31.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:30.288185+0000 mgr.smithi067.srbzpc (mgr.14156) 66 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 -- 172.21.15.169:0/2037889137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 msgr2=0x7fc4cc0f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 --2- 172.21.15.169:0/2037889137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc0f4c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc4b4009fb0 tx=0x7fc4b4009b30).stop 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 -- 172.21.15.169:0/2037889137 shutdown_connections 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 --2- 172.21.15.169:0/2037889137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc0f4c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 -- 172.21.15.169:0/2037889137 >> 172.21.15.169:0/2037889137 conn(0x7fc4cc0f0360 msgr2=0x7fc4cc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 -- 172.21.15.169:0/2037889137 shutdown_connections 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.800+0000 7fc4d11a8700 1 -- 172.21.15.169:0/2037889137 wait complete. 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.801+0000 7fc4d11a8700 1 Processor -- start 2024-01-11T05:05:32.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.801+0000 7fc4d11a8700 1 -- start start 2024-01-11T05:05:32.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4d11a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:32.803 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4d11a8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc106cb0 con 0x7fc4cc0f4840 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4cad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4cad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40100/0 (socket says 172.21.15.169:40100) 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4cad9d700 1 -- 172.21.15.169:0/3637466307 learned_addr learned my addr 172.21.15.169:0/3637466307 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4cad9d700 1 -- 172.21.15.169:0/3637466307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4005040 con 0x7fc4cc0f4840 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4cad9d700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc1067c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc4b40089f0 tx=0x7fc4b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.802+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4b4009320 con 0x7fc4cc0f4840 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.803+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc4b4013ec0 con 0x7fc4cc0f4840 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.803+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc106eb0 con 0x7fc4cc0f4840 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.803+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4b4014d30 con 0x7fc4cc0f4840 2024-01-11T05:05:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.803+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc107340 con 0x7fc4cc0f4840 2024-01-11T05:05:32.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.803+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc4b40095f0 con 0x7fc4cc0f4840 2024-01-11T05:05:32.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.803+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc045bb0 con 0x7fc4cc0f4840 2024-01-11T05:05:32.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.804+0000 7fc4c37fe700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc4b8031090 0x7fc4b8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:32.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.804+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc4b40424a0 con 0x7fc4cc0f4840 2024-01-11T05:05:32.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.804+0000 7fc4ca59c700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc4b8031090 0x7fc4b8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:32.806 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.804+0000 7fc4ca59c700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc4b8031090 0x7fc4b8033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc4bc0096a0 tx=0x7fc4bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:32.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:32.808+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc4b401d7f0 con 0x7fc4cc0f4840 2024-01-11T05:05:33.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.022+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4cc1081f0 con 0x7fc4cc0f4840 2024-01-11T05:05:33.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.022+0000 7fc4c37fe700 1 -- 172.21.15.169:0/3637466307 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4b401d7f0 con 0x7fc4cc0f4840 2024-01-11T05:05:33.023 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:33.023 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:33.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc4b8031090 msgr2=0x7fc4b8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:33.025 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc4b8031090 0x7fc4b8033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc4bc0096a0 tx=0x7fc4bc006b40).stop 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 msgr2=0x7fc4cc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc1067c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc4b40089f0 tx=0x7fc4b400b040).stop 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 shutdown_connections 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc4b8031090 0x7fc4b8033550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 --2- 172.21.15.169:0/3637466307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4cc0f4840 0x7fc4cc1067c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 >> 172.21.15.169:0/3637466307 conn(0x7fc4cc0f0360 msgr2=0x7fc4cc0f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 shutdown_connections 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:33.024+0000 7fc4d11a8700 1 -- 172.21.15.169:0/3637466307 wait complete. 2024-01-11T05:05:33.026 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:32.288520+0000 mgr.smithi067.srbzpc (mgr.14156) 67 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:33.023691+0000 mon.smithi067 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.169:0/3637466307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:34.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:34.418 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:34.288802+0000 mgr.smithi067.srbzpc (mgr.14156) 68 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:35.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- 172.21.15.169:0/757108732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 msgr2=0x7f80600f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:35.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 --2- 172.21.15.169:0/757108732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80600f8fb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8050009fb0 tx=0x7f8050009b30).stop 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- 172.21.15.169:0/757108732 shutdown_connections 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 --2- 172.21.15.169:0/757108732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80600f8fb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- 172.21.15.169:0/757108732 >> 172.21.15.169:0/757108732 conn(0x7f80600f0a30 msgr2=0x7f80600f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- 172.21.15.169:0/757108732 shutdown_connections 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- 172.21.15.169:0/757108732 wait complete. 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 Processor -- start 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- start start 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80601065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.596+0000 7f805ffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8060106aa0 con 0x7f8060060360 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.597+0000 7f805effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80601065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.597+0000 7f805effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80601065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40122/0 (socket says 172.21.15.169:40122) 2024-01-11T05:05:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.597+0000 7f805effd700 1 -- 172.21.15.169:0/3227100832 learned_addr learned my addr 172.21.15.169:0/3227100832 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.597+0000 7f805effd700 1 -- 172.21.15.169:0/3227100832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8050005040 con 0x7f8060060360 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.597+0000 7f805effd700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80601065b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8050009f80 tx=0x7f805000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.597+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8050009320 con 0x7f8060060360 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.598+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8060106ca0 con 0x7f8060060360 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.598+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8050013680 con 0x7f8060060360 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.598+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f805001eda0 con 0x7f8060060360 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.598+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8060107130 con 0x7f8060060360 2024-01-11T05:05:35.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.598+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f805001e420 con 0x7f8060060360 2024-01-11T05:05:35.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.599+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8060045bb0 con 0x7f8060060360 2024-01-11T05:05:35.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.599+0000 7f80477fe700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8048031090 0x7f8048033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:35.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.599+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8050042420 con 0x7f8060060360 2024-01-11T05:05:35.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.599+0000 7f805e7fc700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8048031090 0x7f8048033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:35.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.599+0000 7f805e7fc700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8048031090 0x7f8048033550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:35.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.604+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f805001d380 con 0x7f8060060360 2024-01-11T05:05:35.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.813+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8060107f80 con 0x7f8060060360 2024-01-11T05:05:35.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.813+0000 7f80477fe700 1 -- 172.21.15.169:0/3227100832 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f805001cb10 con 0x7f8060060360 2024-01-11T05:05:35.814 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:35.815 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8048031090 msgr2=0x7f8048033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8048031090 0x7f8048033550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).stop 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 msgr2=0x7f80601065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80601065b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8050009f80 tx=0x7f805000b040).stop 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 shutdown_connections 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8048031090 0x7f8048033550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:35.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 --2- 172.21.15.169:0/3227100832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8060060360 0x7f80601065b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:35.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 >> 172.21.15.169:0/3227100832 conn(0x7f80600f0a30 msgr2=0x7f80600f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:35.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 shutdown_connections 2024-01-11T05:05:35.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:35.815+0000 7f805ffff700 1 -- 172.21.15.169:0/3227100832 wait complete. 2024-01-11T05:05:35.817 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:36.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:35.814597+0000 mon.smithi067 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.169:0/3227100832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:37.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:37.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:37.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:36.289072+0000 mgr.smithi067.srbzpc (mgr.14156) 69 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:38.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 -- 172.21.15.169:0/1374379496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 msgr2=0x7ff2bc0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:38.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 --2- 172.21.15.169:0/1374379496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc0f58d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7ff2ac009fb0 tx=0x7ff2ac009b30).stop 2024-01-11T05:05:38.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 -- 172.21.15.169:0/1374379496 shutdown_connections 2024-01-11T05:05:38.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 --2- 172.21.15.169:0/1374379496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc0f58d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:38.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 -- 172.21.15.169:0/1374379496 >> 172.21.15.169:0/1374379496 conn(0x7ff2bc0f0c10 msgr2=0x7ff2bc0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:38.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 -- 172.21.15.169:0/1374379496 shutdown_connections 2024-01-11T05:05:38.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.367+0000 7ff2c12c0700 1 -- 172.21.15.169:0/1374379496 wait complete. 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2c12c0700 1 Processor -- start 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2c12c0700 1 -- start start 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2c12c0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2c12c0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc106cb0 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2baffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2baffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40136/0 (socket says 172.21.15.169:40136) 2024-01-11T05:05:38.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2baffd700 1 -- 172.21.15.169:0/2871774390 learned_addr learned my addr 172.21.15.169:0/2871774390 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.368+0000 7ff2baffd700 1 -- 172.21.15.169:0/2871774390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac005040 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.370+0000 7ff2baffd700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc1067c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff2ac000c00 tx=0x7ff2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.370+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ac0095f0 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.370+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc106eb0 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.370+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff2ac007e50 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.370+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2ac014e70 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.370+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc107340 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.371+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff2ac025070 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.371+0000 7ff2a37fe700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff2a4031090 0x7ff2a4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.371+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff2ac041de0 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.371+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc045bb0 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.371+0000 7ff2ba7fc700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff2a4031090 0x7ff2a4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:38.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.371+0000 7ff2ba7fc700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff2a4031090 0x7ff2a4033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff2b00096a0 tx=0x7ff2b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:38.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.376+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff2ac026350 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.586+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2bc0fb0f0 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.586 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.587+0000 7ff2a37fe700 1 -- 172.21.15.169:0/2871774390 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2ac02d090 con 0x7ff2bc0f54b0 2024-01-11T05:05:38.586 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:38.587 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff2a4031090 msgr2=0x7ff2a4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff2a4031090 0x7ff2a4033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff2b00096a0 tx=0x7ff2b0006b40).stop 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 msgr2=0x7ff2bc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc1067c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff2ac000c00 tx=0x7ff2ac00b040).stop 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 shutdown_connections 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff2a4031090 0x7ff2a4033550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:38.588 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 --2- 172.21.15.169:0/2871774390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc0f54b0 0x7ff2bc1067c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:38.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 >> 172.21.15.169:0/2871774390 conn(0x7ff2bc0f0c10 msgr2=0x7ff2bc05ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:38.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 shutdown_connections 2024-01-11T05:05:38.589 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:38.588+0000 7ff2c12c0700 1 -- 172.21.15.169:0/2871774390 wait complete. 2024-01-11T05:05:38.589 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:38.289343+0000 mgr.smithi067.srbzpc (mgr.14156) 70 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:38.586958+0000 mon.smithi067 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.169:0/2871774390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:39.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:39.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 -- 172.21.15.169:0/1610996848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 msgr2=0x7f401c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 --2- 172.21.15.169:0/1610996848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c0f58a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f400c009fb0 tx=0x7f400c009b30).stop 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 -- 172.21.15.169:0/1610996848 shutdown_connections 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 --2- 172.21.15.169:0/1610996848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c0f58a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 -- 172.21.15.169:0/1610996848 >> 172.21.15.169:0/1610996848 conn(0x7f401c0f0be0 msgr2=0x7f401c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 -- 172.21.15.169:0/1610996848 shutdown_connections 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.084+0000 7f40234d4700 1 -- 172.21.15.169:0/1610996848 wait complete. 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f40234d4700 1 Processor -- start 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f40234d4700 1 -- start start 2024-01-11T05:05:41.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f40234d4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f40234d4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c105be0 con 0x7f401c0f5480 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f4021270700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f4021270700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46418/0 (socket says 172.21.15.169:46418) 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f4021270700 1 -- 172.21.15.169:0/2360053866 learned_addr learned my addr 172.21.15.169:0/2360053866 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.085+0000 7f4021270700 1 -- 172.21.15.169:0/2360053866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c005040 con 0x7f401c0f5480 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.086+0000 7f4021270700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c1056f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f400c0089f0 tx=0x7f400c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.086+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400c009320 con 0x7f401c0f5480 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.086+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c105de0 con 0x7f401c0f5480 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.086+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f400c013ec0 con 0x7f401c0f5480 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.086+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400c014d30 con 0x7f401c0f5480 2024-01-11T05:05:41.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.086+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c106270 con 0x7f401c0f5480 2024-01-11T05:05:41.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.087+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f400c0095f0 con 0x7f401c0f5480 2024-01-11T05:05:41.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.087+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401c045bb0 con 0x7f401c0f5480 2024-01-11T05:05:41.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.087+0000 7f4011ffb700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4008031090 0x7f4008033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:41.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.087+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f400c0424a0 con 0x7f401c0f5480 2024-01-11T05:05:41.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.087+0000 7f4020a6f700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4008031090 0x7f4008033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:41.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.088+0000 7f4020a6f700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4008031090 0x7f4008033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f40180096a0 tx=0x7f4018006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:41.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.092+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f400c0268c0 con 0x7f401c0f5480 2024-01-11T05:05:41.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.298+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f401c0faf70 con 0x7f401c0f5480 2024-01-11T05:05:41.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.298+0000 7f4011ffb700 1 -- 172.21.15.169:0/2360053866 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f400c01d7f0 con 0x7f401c0f5480 2024-01-11T05:05:41.298 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:41.299 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:41.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4008031090 msgr2=0x7f4008033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:41.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4008031090 0x7f4008033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f40180096a0 tx=0x7f4018006b40).stop 2024-01-11T05:05:41.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 msgr2=0x7f401c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:41.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c1056f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f400c0089f0 tx=0x7f400c00b040).stop 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 shutdown_connections 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4008031090 0x7f4008033550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 --2- 172.21.15.169:0/2360053866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f401c0f5480 0x7f401c1056f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 >> 172.21.15.169:0/2360053866 conn(0x7f401c0f0be0 msgr2=0x7f401c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 shutdown_connections 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:41.300+0000 7f40234d4700 1 -- 172.21.15.169:0/2360053866 wait complete. 2024-01-11T05:05:41.301 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:40.289767+0000 mgr.smithi067.srbzpc (mgr.14156) 71 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:41.298649+0000 mon.smithi067 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.169:0/2360053866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:42.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:42.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:42.290170+0000 mgr.smithi067.srbzpc (mgr.14156) 72 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 -- 172.21.15.169:0/2353581585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 msgr2=0x7f07a40f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 --2- 172.21.15.169:0/2353581585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a40f58d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0794009fb0 tx=0x7f0794009b30).stop 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 -- 172.21.15.169:0/2353581585 shutdown_connections 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 --2- 172.21.15.169:0/2353581585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a40f58d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 -- 172.21.15.169:0/2353581585 >> 172.21.15.169:0/2353581585 conn(0x7f07a40f0c10 msgr2=0x7f07a40f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 -- 172.21.15.169:0/2353581585 shutdown_connections 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.862+0000 7f07ab41d700 1 -- 172.21.15.169:0/2353581585 wait complete. 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07ab41d700 1 Processor -- start 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07ab41d700 1 -- start start 2024-01-11T05:05:43.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07ab41d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a41094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07ab41d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4060300 con 0x7f07a40f54b0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07a91b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a41094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07a91b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a41094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46432/0 (socket says 172.21.15.169:46432) 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07a91b9700 1 -- 172.21.15.169:0/3391748596 learned_addr learned my addr 172.21.15.169:0/3391748596 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07a91b9700 1 -- 172.21.15.169:0/3391748596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794005040 con 0x7f07a40f54b0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.863+0000 7f07a91b9700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a41094c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f07940089f0 tx=0x7f079400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.864+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0794009320 con 0x7f07a40f54b0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.864+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4060500 con 0x7f07a40f54b0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.864+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0794013ec0 con 0x7f07a40f54b0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.864+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0794014d30 con 0x7f07a40f54b0 2024-01-11T05:05:43.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.864+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a4109b70 con 0x7f07a40f54b0 2024-01-11T05:05:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.865+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f07940263a0 con 0x7f07a40f54b0 2024-01-11T05:05:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.865+0000 7f0799ffb700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0790031090 0x7f0790033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.865+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0794042460 con 0x7f07a40f54b0 2024-01-11T05:05:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.865+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a40591e0 con 0x7f07a40f54b0 2024-01-11T05:05:43.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.865+0000 7f07a89b8700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0790031090 0x7f0790033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:43.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.865+0000 7f07a89b8700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0790031090 0x7f0790033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f07a00096a0 tx=0x7f07a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:43.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:43.870+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f07940266d0 con 0x7f07a40f54b0 2024-01-11T05:05:44.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.081+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07a4045bb0 con 0x7f07a40f54b0 2024-01-11T05:05:44.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.082+0000 7f0799ffb700 1 -- 172.21.15.169:0/3391748596 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f079402b020 con 0x7f07a40f54b0 2024-01-11T05:05:44.082 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:44.082 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:44.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0790031090 msgr2=0x7f0790033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:44.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0790031090 0x7f0790033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f07a00096a0 tx=0x7f07a0006b40).stop 2024-01-11T05:05:44.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 msgr2=0x7f07a41094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:44.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a41094c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f07940089f0 tx=0x7f079400b040).stop 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 shutdown_connections 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0790031090 0x7f0790033550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 --2- 172.21.15.169:0/3391748596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40f54b0 0x7f07a41094c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 >> 172.21.15.169:0/3391748596 conn(0x7f07a40f0c10 msgr2=0x7f07a40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 shutdown_connections 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:44.083+0000 7f07ab41d700 1 -- 172.21.15.169:0/3391748596 wait complete. 2024-01-11T05:05:44.084 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:45.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:45.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:44.082138+0000 mon.smithi067 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.169:0/3391748596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:44.290431+0000 mgr.smithi067.srbzpc (mgr.14156) 73 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.617+0000 7f877afac700 1 -- 172.21.15.169:0/3656083523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 msgr2=0x7f87740f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.617+0000 7f877afac700 1 --2- 172.21.15.169:0/3656083523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87740f58a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8768009fb0 tx=0x7f8768009b30).stop 2024-01-11T05:05:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.617+0000 7f877afac700 1 -- 172.21.15.169:0/3656083523 shutdown_connections 2024-01-11T05:05:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.617+0000 7f877afac700 1 --2- 172.21.15.169:0/3656083523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87740f58a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:46.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.617+0000 7f877afac700 1 -- 172.21.15.169:0/3656083523 >> 172.21.15.169:0/3656083523 conn(0x7f87740f0be0 msgr2=0x7f87740f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.617+0000 7f877afac700 1 -- 172.21.15.169:0/3656083523 shutdown_connections 2024-01-11T05:05:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.618+0000 7f877afac700 1 -- 172.21.15.169:0/3656083523 wait complete. 2024-01-11T05:05:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.618+0000 7f877afac700 1 Processor -- start 2024-01-11T05:05:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.618+0000 7f877afac700 1 -- start start 2024-01-11T05:05:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f877afac700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87741056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:46.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f877afac700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8774105be0 con 0x7f87740f5480 2024-01-11T05:05:46.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f8778d48700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87741056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f8778d48700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87741056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46452/0 (socket says 172.21.15.169:46452) 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f8778d48700 1 -- 172.21.15.169:0/218079572 learned_addr learned my addr 172.21.15.169:0/218079572 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f8778d48700 1 -- 172.21.15.169:0/218079572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8768005040 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.619+0000 7f8778d48700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87741056f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f87680089f0 tx=0x7f876800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.620+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8768009320 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.620+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8774105de0 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.620+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8768013ec0 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.620+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8768014d30 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.620+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8774106270 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.620+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8768014420 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.621+0000 7f87717fa700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8764031090 0x7f8764033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.621+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8774045bb0 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.621+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8768044270 con 0x7f87740f5480 2024-01-11T05:05:46.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.621+0000 7f8773fff700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8764031090 0x7f8764033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:46.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.621+0000 7f8773fff700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8764031090 0x7f8764033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f87600096a0 tx=0x7f8760006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:46.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.626+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8768014750 con 0x7f87740f5480 2024-01-11T05:05:46.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.836+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8774107070 con 0x7f87740f5480 2024-01-11T05:05:46.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.837+0000 7f87717fa700 1 -- 172.21.15.169:0/218079572 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f876802d020 con 0x7f87740f5480 2024-01-11T05:05:46.838 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:46.838 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8764031090 msgr2=0x7f8764033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8764031090 0x7f8764033550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f87600096a0 tx=0x7f8760006b40).stop 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 msgr2=0x7f87741056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87741056f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f87680089f0 tx=0x7f876800b040).stop 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 shutdown_connections 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8764031090 0x7f8764033550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 --2- 172.21.15.169:0/218079572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87740f5480 0x7f87741056f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 >> 172.21.15.169:0/218079572 conn(0x7f87740f0be0 msgr2=0x7f87740f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 shutdown_connections 2024-01-11T05:05:46.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:46.839+0000 7f877afac700 1 -- 172.21.15.169:0/218079572 wait complete. 2024-01-11T05:05:46.841 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:46.290662+0000 mgr.smithi067.srbzpc (mgr.14156) 74 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:46.838562+0000 mon.smithi067 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.169:0/218079572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:48.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:48.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:49.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 -- 172.21.15.169:0/3645363307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 msgr2=0x7f06600f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:49.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 --2- 172.21.15.169:0/3645363307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f06600f56f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f0654009fb0 tx=0x7f0654009b30).stop 2024-01-11T05:05:49.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 -- 172.21.15.169:0/3645363307 shutdown_connections 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 --2- 172.21.15.169:0/3645363307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f06600f56f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 -- 172.21.15.169:0/3645363307 >> 172.21.15.169:0/3645363307 conn(0x7f06600f0a30 msgr2=0x7f06600f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 -- 172.21.15.169:0/3645363307 shutdown_connections 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 -- 172.21.15.169:0/3645363307 wait complete. 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.326+0000 7f065f59e700 1 Processor -- start 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065f59e700 1 -- start start 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065f59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f0660105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:49.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065f59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660105a00 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f0660105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f0660105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46482/0 (socket says 172.21.15.169:46482) 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065e59c700 1 -- 172.21.15.169:0/3379365530 learned_addr learned my addr 172.21.15.169:0/3379365530 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065e59c700 1 -- 172.21.15.169:0/3379365530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654005040 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.327+0000 7f065e59c700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f0660105510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f06540089f0 tx=0x7f065400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.328+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0654009320 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.328+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0660105c00 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.328+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0654013ec0 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.328+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0654014d30 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.328+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0660106090 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.329+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f06540095f0 con 0x7f06600f52d0 2024-01-11T05:05:49.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.329+0000 7f064effd700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0648031090 0x7f0648033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.329+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0660045bb0 con 0x7f06600f52d0 2024-01-11T05:05:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.329+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f06540423e0 con 0x7f06600f52d0 2024-01-11T05:05:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.329+0000 7f065dd9b700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0648031090 0x7f0648033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:49.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.329+0000 7f065dd9b700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0648031090 0x7f0648033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f06500096a0 tx=0x7f0650006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:49.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.334+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f065401dd70 con 0x7f06600f52d0 2024-01-11T05:05:49.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.542+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0660106f10 con 0x7f06600f52d0 2024-01-11T05:05:49.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.543+0000 7f064effd700 1 -- 172.21.15.169:0/3379365530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f065403f030 con 0x7f06600f52d0 2024-01-11T05:05:49.543 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:49.543 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:49.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0648031090 msgr2=0x7f0648033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0648031090 0x7f0648033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f06500096a0 tx=0x7f0650006b40).stop 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 msgr2=0x7f0660105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f0660105510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f06540089f0 tx=0x7f065400b040).stop 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 shutdown_connections 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0648031090 0x7f0648033550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 --2- 172.21.15.169:0/3379365530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06600f52d0 0x7f0660105510 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 >> 172.21.15.169:0/3379365530 conn(0x7f06600f0a30 msgr2=0x7f06600f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 shutdown_connections 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:49.544+0000 7f065f59e700 1 -- 172.21.15.169:0/3379365530 wait complete. 2024-01-11T05:05:49.546 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:48.290998+0000 mgr.smithi067.srbzpc (mgr.14156) 75 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:50.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:49.543829+0000 mon.smithi067 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.169:0/3379365530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:50.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:50.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:50.291384+0000 mgr.smithi067.srbzpc (mgr.14156) 76 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:52.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 -- 172.21.15.169:0/4258339442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 msgr2=0x7fefd40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:52.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 --2- 172.21.15.169:0/4258339442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd40f69b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fefd0004760 tx=0x7fefd0009ad0).stop 2024-01-11T05:05:52.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 -- 172.21.15.169:0/4258339442 shutdown_connections 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 --2- 172.21.15.169:0/4258339442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd40f69b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 -- 172.21.15.169:0/4258339442 >> 172.21.15.169:0/4258339442 conn(0x7fefd40f1cd0 msgr2=0x7fefd40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 -- 172.21.15.169:0/4258339442 shutdown_connections 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.018+0000 7fefdbb68700 1 -- 172.21.15.169:0/4258339442 wait complete. 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefdbb68700 1 Processor -- start 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefdbb68700 1 -- start start 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefdbb68700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefdbb68700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd4106cb0 con 0x7fefd40f6590 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefd9904700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefd9904700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35034/0 (socket says 172.21.15.169:35034) 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefd9904700 1 -- 172.21.15.169:0/2192503801 learned_addr learned my addr 172.21.15.169:0/2192503801 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefd9904700 1 -- 172.21.15.169:0/2192503801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd0009d70 con 0x7fefd40f6590 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.019+0000 7fefd9904700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd41067c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fefd0013f10 tx=0x7fefd0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:52.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.020+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd000e040 con 0x7fefd40f6590 2024-01-11T05:05:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.020+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fefd0007df0 con 0x7fefd40f6590 2024-01-11T05:05:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.020+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd4106eb0 con 0x7fefd40f6590 2024-01-11T05:05:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.020+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd0014d60 con 0x7fefd40f6590 2024-01-11T05:05:52.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.020+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd4107340 con 0x7fefd40f6590 2024-01-11T05:05:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.020+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fefd0014520 con 0x7fefd40f6590 2024-01-11T05:05:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.021+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd4045bb0 con 0x7fefd40f6590 2024-01-11T05:05:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.021+0000 7fefca7fc700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fefc0031090 0x7fefc0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.021+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fefd00424c0 con 0x7fefd40f6590 2024-01-11T05:05:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.021+0000 7fefd9103700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fefc0031090 0x7fefc0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:52.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.021+0000 7fefd9103700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fefc0031090 0x7fefc0033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fefc40096a0 tx=0x7fefc4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:52.026 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.025+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fefd0007960 con 0x7fefd40f6590 2024-01-11T05:05:52.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.235+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefd41081f0 con 0x7fefd40f6590 2024-01-11T05:05:52.236 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.236+0000 7fefca7fc700 1 -- 172.21.15.169:0/2192503801 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefd00262f0 con 0x7fefd40f6590 2024-01-11T05:05:52.237 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:52.237 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fefc0031090 msgr2=0x7fefc0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fefc0031090 0x7fefc0033550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fefc40096a0 tx=0x7fefc4006b40).stop 2024-01-11T05:05:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 msgr2=0x7fefd41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd41067c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fefd0013f10 tx=0x7fefd0009ad0).stop 2024-01-11T05:05:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 shutdown_connections 2024-01-11T05:05:52.238 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fefc0031090 0x7fefc0033550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:52.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 --2- 172.21.15.169:0/2192503801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd40f6590 0x7fefd41067c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:52.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 >> 172.21.15.169:0/2192503801 conn(0x7fefd40f1cd0 msgr2=0x7fefd40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:52.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 shutdown_connections 2024-01-11T05:05:52.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:52.237+0000 7fefdbb68700 1 -- 172.21.15.169:0/2192503801 wait complete. 2024-01-11T05:05:52.239 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:52.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:52.236993+0000 mon.smithi067 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.169:0/2192503801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:53.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:53.614 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:53.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:52.291705+0000 mgr.smithi067.srbzpc (mgr.14156) 77 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:54.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 -- 172.21.15.169:0/2071374716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 msgr2=0x7fc9400f5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:54.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 --2- 172.21.15.169:0/2071374716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9400f5ce0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fc934009fb0 tx=0x7fc934009b30).stop 2024-01-11T05:05:54.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 -- 172.21.15.169:0/2071374716 shutdown_connections 2024-01-11T05:05:54.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 --2- 172.21.15.169:0/2071374716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9400f5ce0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:54.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 -- 172.21.15.169:0/2071374716 >> 172.21.15.169:0/2071374716 conn(0x7fc9400f01d0 msgr2=0x7fc9400f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:54.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 -- 172.21.15.169:0/2071374716 shutdown_connections 2024-01-11T05:05:54.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 -- 172.21.15.169:0/2071374716 wait complete. 2024-01-11T05:05:54.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 Processor -- start 2024-01-11T05:05:54.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.726+0000 7fc945945700 1 -- start start 2024-01-11T05:05:54.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc945945700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9401054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:54.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc945945700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9401059d0 con 0x7fc9400f58c0 2024-01-11T05:05:54.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc944943700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9401054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc944943700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9401054e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35054/0 (socket says 172.21.15.169:35054) 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc944943700 1 -- 172.21.15.169:0/244324008 learned_addr learned my addr 172.21.15.169:0/244324008 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc944943700 1 -- 172.21.15.169:0/244324008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc934005040 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc944943700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9401054e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc9340089f0 tx=0x7fc93400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc934009320 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc940105bd0 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc934013ec0 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc934014d30 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.727+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc940106060 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.728+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc9340095f0 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.728+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc940045bb0 con 0x7fc9400f58c0 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.728+0000 7fc93d7fa700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc930031090 0x7fc930033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:54.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.728+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc9340424a0 con 0x7fc9400f58c0 2024-01-11T05:05:54.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.729+0000 7fc93ffff700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc930031090 0x7fc930033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:54.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.729+0000 7fc93ffff700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc930031090 0x7fc930033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fc92c0096a0 tx=0x7fc92c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:54.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.733+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc93401d7f0 con 0x7fc9400f58c0 2024-01-11T05:05:54.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.939+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc940106f10 con 0x7fc9400f58c0 2024-01-11T05:05:54.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.940+0000 7fc93d7fa700 1 -- 172.21.15.169:0/244324008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc93401d7f0 con 0x7fc9400f58c0 2024-01-11T05:05:54.941 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:54.941 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:54.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc930031090 msgr2=0x7fc930033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc930031090 0x7fc930033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fc92c0096a0 tx=0x7fc92c006b40).stop 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 msgr2=0x7fc9401054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9401054e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc9340089f0 tx=0x7fc93400b040).stop 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 shutdown_connections 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc930031090 0x7fc930033550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 --2- 172.21.15.169:0/244324008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9400f58c0 0x7fc9401054e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 >> 172.21.15.169:0/244324008 conn(0x7fc9400f01d0 msgr2=0x7fc9400f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 shutdown_connections 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:54.942+0000 7fc945945700 1 -- 172.21.15.169:0/244324008 wait complete. 2024-01-11T05:05:54.943 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:54.292111+0000 mgr.smithi067.srbzpc (mgr.14156) 78 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:54.941233+0000 mon.smithi067 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.169:0/244324008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:56.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:56.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 -- 172.21.15.169:0/4151448532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 msgr2=0x7f5c8c0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 --2- 172.21.15.169:0/4151448532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c0f58d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5c7c009fb0 tx=0x7f5c7c009b30).stop 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 -- 172.21.15.169:0/4151448532 shutdown_connections 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 --2- 172.21.15.169:0/4151448532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c0f58d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 -- 172.21.15.169:0/4151448532 >> 172.21.15.169:0/4151448532 conn(0x7f5c8c0f0c10 msgr2=0x7f5c8c0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 -- 172.21.15.169:0/4151448532 shutdown_connections 2024-01-11T05:05:57.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 -- 172.21.15.169:0/4151448532 wait complete. 2024-01-11T05:05:57.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 Processor -- start 2024-01-11T05:05:57.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.465+0000 7f5c9201f700 1 -- start start 2024-01-11T05:05:57.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c9201f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c1056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:57.467 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c9201f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c105b90 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c8b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c1056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:57.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c8b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c1056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35068/0 (socket says 172.21.15.169:35068) 2024-01-11T05:05:57.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c8b7fe700 1 -- 172.21.15.169:0/1139656350 learned_addr learned my addr 172.21.15.169:0/1139656350 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c8b7fe700 1 -- 172.21.15.169:0/1139656350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c005040 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.466+0000 7f5c8b7fe700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c1056a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5c7c0089f0 tx=0x7f5c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.467+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c7c009320 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.467+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c105d90 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.467+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5c7c013ec0 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.467+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c7c014d30 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.467+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c106220 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.467+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5c7c014420 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.468+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c045bb0 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.468+0000 7f5c83fff700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5c78031090 0x7f5c78033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.468+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5c7c0424c0 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.468+0000 7f5c8affd700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5c78031090 0x7f5c78033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:05:57.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.468+0000 7f5c8affd700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5c78031090 0x7f5c78033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5c740096a0 tx=0x7f5c74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:05:57.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.473+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5c7c01d980 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.682+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c8c1070d0 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.682+0000 7f5c83fff700 1 -- 172.21.15.169:0/1139656350 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c7c01d380 con 0x7f5c8c0f54b0 2024-01-11T05:05:57.684 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:05:57.684 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:05:57.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.684+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5c78031090 msgr2=0x7f5c78033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:57.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.684+0000 7f5c9201f700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5c78031090 0x7f5c78033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5c740096a0 tx=0x7f5c74006b40).stop 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.684+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 msgr2=0x7f5c8c1056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.684+0000 7f5c9201f700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c1056a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5c7c0089f0 tx=0x7f5c7c00b040).stop 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.684+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 shutdown_connections 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.684+0000 7f5c9201f700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5c78031090 0x7f5c78033550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.685+0000 7f5c9201f700 1 --2- 172.21.15.169:0/1139656350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c8c0f54b0 0x7f5c8c1056a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.685+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 >> 172.21.15.169:0/1139656350 conn(0x7f5c8c0f0c10 msgr2=0x7f5c8c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.685+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 shutdown_connections 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:05:57.685+0000 7f5c9201f700 1 -- 172.21.15.169:0/1139656350 wait complete. 2024-01-11T05:05:57.686 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:05:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:56.292435+0000 mgr.smithi067.srbzpc (mgr.14156) 79 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:05:58.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:58 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:05:57.683842+0000 mon.smithi067 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.169:0/1139656350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:05:59.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:05:59.056 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:05:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:05:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:05:58.292775+0000 mgr.smithi067.srbzpc (mgr.14156) 80 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 -- 172.21.15.169:0/2559879830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 msgr2=0x7f79a40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 --2- 172.21.15.169:0/2559879830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a40f56f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f7994009fb0 tx=0x7f7994009b30).stop 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 -- 172.21.15.169:0/2559879830 shutdown_connections 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 --2- 172.21.15.169:0/2559879830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a40f56f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 -- 172.21.15.169:0/2559879830 >> 172.21.15.169:0/2559879830 conn(0x7f79a40f0a30 msgr2=0x7f79a40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 -- 172.21.15.169:0/2559879830 shutdown_connections 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 -- 172.21.15.169:0/2559879830 wait complete. 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.147+0000 7f79a359e700 1 Processor -- start 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a359e700 1 -- start start 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a359e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:00.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a359e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a4106af0 con 0x7f79a40f52d0 2024-01-11T05:06:00.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:00.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:50104/0 (socket says 172.21.15.169:50104) 2024-01-11T05:06:00.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a259c700 1 -- 172.21.15.169:0/592262632 learned_addr learned my addr 172.21.15.169:0/592262632 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:00.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a259c700 1 -- 172.21.15.169:0/592262632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994005040 con 0x7f79a40f52d0 2024-01-11T05:06:00.150 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a259c700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a4106600 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f79940089f0 tx=0x7f799400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:00.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7994009320 con 0x7f79a40f52d0 2024-01-11T05:06:00.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a4106cf0 con 0x7f79a40f52d0 2024-01-11T05:06:00.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7994013ec0 con 0x7f79a40f52d0 2024-01-11T05:06:00.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7994014d30 con 0x7f79a40f52d0 2024-01-11T05:06:00.151 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.148+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a4107180 con 0x7f79a40f52d0 2024-01-11T05:06:00.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.149+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f79940095f0 con 0x7f79a40f52d0 2024-01-11T05:06:00.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.149+0000 7f7992ffd700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f798c031090 0x7f798c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:00.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.149+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f79940423e0 con 0x7f79a40f52d0 2024-01-11T05:06:00.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.149+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a4045bb0 con 0x7f79a40f52d0 2024-01-11T05:06:00.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.150+0000 7f79a1d9b700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f798c031090 0x7f798c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:00.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.150+0000 7f79a1d9b700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f798c031090 0x7f798c033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f79980096a0 tx=0x7f7998006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:00.155 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.154+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f799401dd70 con 0x7f79a40f52d0 2024-01-11T05:06:00.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.366+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79a40faeb0 con 0x7f79a40f52d0 2024-01-11T05:06:00.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.367+0000 7f7992ffd700 1 -- 172.21.15.169:0/592262632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f799402b020 con 0x7f79a40f52d0 2024-01-11T05:06:00.368 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:00.368 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f798c031090 msgr2=0x7f798c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f798c031090 0x7f798c033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f79980096a0 tx=0x7f7998006b40).stop 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 msgr2=0x7f79a4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a4106600 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f79940089f0 tx=0x7f799400b040).stop 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 shutdown_connections 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f798c031090 0x7f798c033550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 --2- 172.21.15.169:0/592262632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a40f52d0 0x7f79a4106600 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 >> 172.21.15.169:0/592262632 conn(0x7f79a40f0a30 msgr2=0x7f79a405fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 shutdown_connections 2024-01-11T05:06:00.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:00.369+0000 7f79a359e700 1 -- 172.21.15.169:0/592262632 wait complete. 2024-01-11T05:06:00.371 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:01.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:01.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:01.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:00.293183+0000 mgr.smithi067.srbzpc (mgr.14156) 81 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:00.368278+0000 mon.smithi067 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.169:0/592262632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 -- 172.21.15.169:0/84152683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 msgr2=0x7fbee80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 --2- 172.21.15.169:0/84152683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee80f58a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbed0009fb0 tx=0x7fbed0009b30).stop 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 -- 172.21.15.169:0/84152683 shutdown_connections 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 --2- 172.21.15.169:0/84152683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee80f58a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 -- 172.21.15.169:0/84152683 >> 172.21.15.169:0/84152683 conn(0x7fbee80f0be0 msgr2=0x7fbee80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 -- 172.21.15.169:0/84152683 shutdown_connections 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.829+0000 7fbeedbf4700 1 -- 172.21.15.169:0/84152683 wait complete. 2024-01-11T05:06:02.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.830+0000 7fbeedbf4700 1 Processor -- start 2024-01-11T05:06:02.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.830+0000 7fbeedbf4700 1 -- start start 2024-01-11T05:06:02.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.830+0000 7fbeedbf4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:02.832 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.830+0000 7fbeedbf4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee8060300 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbee77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbee77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:50124/0 (socket says 172.21.15.169:50124) 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbee77fe700 1 -- 172.21.15.169:0/850803341 learned_addr learned my addr 172.21.15.169:0/850803341 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbee77fe700 1 -- 172.21.15.169:0/850803341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed0005040 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbee77fe700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee8109510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fbed0000c00 tx=0x7fbed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbed00095f0 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee8060500 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbed0007e50 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbed0014e70 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.831+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee8109bc0 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.832+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbed00079c0 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.832+0000 7fbedffff700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed4031090 0x7fbed4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.832+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbed0041d10 con 0x7fbee80f5480 2024-01-11T05:06:02.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.832+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee80591e0 con 0x7fbee80f5480 2024-01-11T05:06:02.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.832+0000 7fbee6ffd700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed4031090 0x7fbed4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:02.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.833+0000 7fbee6ffd700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed4031090 0x7fbed4033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbed80096a0 tx=0x7fbed8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:02.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:02.837+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbed001b910 con 0x7fbee80f5480 2024-01-11T05:06:03.049 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.048+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbee8045bb0 con 0x7fbee80f5480 2024-01-11T05:06:03.050 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.049+0000 7fbedffff700 1 -- 172.21.15.169:0/850803341 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbed002c090 con 0x7fbee80f5480 2024-01-11T05:06:03.050 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:03.051 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:03.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed4031090 msgr2=0x7fbed4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:03.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed4031090 0x7fbed4033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbed80096a0 tx=0x7fbed8006b40).stop 2024-01-11T05:06:03.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 msgr2=0x7fbee8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:03.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee8109510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fbed0000c00 tx=0x7fbed000b040).stop 2024-01-11T05:06:03.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 shutdown_connections 2024-01-11T05:06:03.052 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed4031090 0x7fbed4033550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:03.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 --2- 172.21.15.169:0/850803341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f5480 0x7fbee8109510 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:03.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 >> 172.21.15.169:0/850803341 conn(0x7fbee80f0be0 msgr2=0x7fbee80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:03.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 shutdown_connections 2024-01-11T05:06:03.053 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:03.051+0000 7fbeedbf4700 1 -- 172.21.15.169:0/850803341 wait complete. 2024-01-11T05:06:03.053 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:02.293583+0000 mgr.smithi067.srbzpc (mgr.14156) 82 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:03.050547+0000 mon.smithi067 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.169:0/850803341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:04.414 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:04.414 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:04.293862+0000 mgr.smithi067.srbzpc (mgr.14156) 83 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:05.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 -- 172.21.15.169:0/3200867016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 msgr2=0x7f97940f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 --2- 172.21.15.169:0/3200867016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f97940f69b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9784009fb0 tx=0x7f9784009b30).stop 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 -- 172.21.15.169:0/3200867016 shutdown_connections 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 --2- 172.21.15.169:0/3200867016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f97940f69b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 -- 172.21.15.169:0/3200867016 >> 172.21.15.169:0/3200867016 conn(0x7f97940f1d10 msgr2=0x7f97940f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 -- 172.21.15.169:0/3200867016 shutdown_connections 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.566+0000 7f979c60f700 1 -- 172.21.15.169:0/3200867016 wait complete. 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979c60f700 1 Processor -- start 2024-01-11T05:06:05.568 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979c60f700 1 -- start start 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979c60f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f9794106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979c60f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9794106cf0 con 0x7f97940f6590 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979a3ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f9794106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979a3ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f9794106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:50144/0 (socket says 172.21.15.169:50144) 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.567+0000 7f979a3ab700 1 -- 172.21.15.169:0/1420170905 learned_addr learned my addr 172.21.15.169:0/1420170905 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f979a3ab700 1 -- 172.21.15.169:0/1420170905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784005040 con 0x7f97940f6590 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f979a3ab700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f9794106800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f97840089f0 tx=0x7f978400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9784009320 con 0x7f97940f6590 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9794106ef0 con 0x7f97940f6590 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9784013ec0 con 0x7f97940f6590 2024-01-11T05:06:05.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9784014d30 con 0x7f97940f6590 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.568+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9794107380 con 0x7f97940f6590 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.569+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f97840095f0 con 0x7f97940f6590 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.569+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9794045bb0 con 0x7f97940f6590 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.569+0000 7f978affd700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9780031090 0x7f9780033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.569+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f97840424a0 con 0x7f97940f6590 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.569+0000 7f9799baa700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9780031090 0x7f9780033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:05.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.570+0000 7f9799baa700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9780031090 0x7f9780033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f97900096a0 tx=0x7f9790006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:05.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.574+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f978401d980 con 0x7f97940f6590 2024-01-11T05:06:05.781 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.781+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97940fc080 con 0x7f97940f6590 2024-01-11T05:06:05.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.781+0000 7f978affd700 1 -- 172.21.15.169:0/1420170905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f978401d380 con 0x7f97940f6590 2024-01-11T05:06:05.782 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:05.782 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9780031090 msgr2=0x7f9780033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9780031090 0x7f9780033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f97900096a0 tx=0x7f9790006b40).stop 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 msgr2=0x7f9794106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f9794106800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f97840089f0 tx=0x7f978400b040).stop 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 shutdown_connections 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9780031090 0x7f9780033550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 --2- 172.21.15.169:0/1420170905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97940f6590 0x7f9794106800 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 >> 172.21.15.169:0/1420170905 conn(0x7f97940f1d10 msgr2=0x7f97940f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:05.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 shutdown_connections 2024-01-11T05:06:05.785 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:05.783+0000 7f979c60f700 1 -- 172.21.15.169:0/1420170905 wait complete. 2024-01-11T05:06:05.785 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:06.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:06 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:05.782754+0000 mon.smithi067 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.169:0/1420170905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:07.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:07.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:07.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:06.294160+0000 mgr.smithi067.srbzpc (mgr.14156) 84 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.266+0000 7f49cf7d7700 1 -- 172.21.15.169:0/2637297500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 msgr2=0x7f49c80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.266+0000 7f49cf7d7700 1 --2- 172.21.15.169:0/2637297500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c80f58a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f49c4009fb0 tx=0x7f49c4009b30).stop 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.266+0000 7f49cf7d7700 1 -- 172.21.15.169:0/2637297500 shutdown_connections 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.266+0000 7f49cf7d7700 1 --2- 172.21.15.169:0/2637297500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c80f58a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.266+0000 7f49cf7d7700 1 -- 172.21.15.169:0/2637297500 >> 172.21.15.169:0/2637297500 conn(0x7f49c80f0be0 msgr2=0x7f49c80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.266+0000 7f49cf7d7700 1 -- 172.21.15.169:0/2637297500 shutdown_connections 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.267+0000 7f49cf7d7700 1 -- 172.21.15.169:0/2637297500 wait complete. 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.267+0000 7f49cf7d7700 1 Processor -- start 2024-01-11T05:06:08.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.267+0000 7f49cf7d7700 1 -- start start 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.267+0000 7f49cf7d7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c81056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.267+0000 7f49cf7d7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8105be0 con 0x7f49c80f5480 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49cd573700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c81056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49cd573700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c81056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:50162/0 (socket says 172.21.15.169:50162) 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49cd573700 1 -- 172.21.15.169:0/3555379128 learned_addr learned my addr 172.21.15.169:0/3555379128 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49cd573700 1 -- 172.21.15.169:0/3555379128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c4005040 con 0x7f49c80f5480 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49cd573700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c81056f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f49c40089f0 tx=0x7f49c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49c4009320 con 0x7f49c80f5480 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f49c4013ec0 con 0x7f49c80f5480 2024-01-11T05:06:08.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c8105de0 con 0x7f49c80f5480 2024-01-11T05:06:08.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.268+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49c4014d30 con 0x7f49c80f5480 2024-01-11T05:06:08.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.269+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c8106270 con 0x7f49c80f5480 2024-01-11T05:06:08.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.269+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f49c40263a0 con 0x7f49c80f5480 2024-01-11T05:06:08.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.269+0000 7f49bdffb700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f49b4031090 0x7f49b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:08.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.270+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f49c4042420 con 0x7f49c80f5480 2024-01-11T05:06:08.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.270+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c8045bb0 con 0x7f49c80f5480 2024-01-11T05:06:08.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.270+0000 7f49ccd72700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f49b4031090 0x7f49b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:08.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.270+0000 7f49ccd72700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f49b4031090 0x7f49b4033550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f49b80096a0 tx=0x7f49b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:08.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.274+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f49c401cb10 con 0x7f49c80f5480 2024-01-11T05:06:08.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.487+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49c80fb060 con 0x7f49c80f5480 2024-01-11T05:06:08.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.488+0000 7f49bdffb700 1 -- 172.21.15.169:0/3555379128 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49c402b020 con 0x7f49c80f5480 2024-01-11T05:06:08.488 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:08.489 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f49b4031090 msgr2=0x7f49b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f49b4031090 0x7f49b4033550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f49b80096a0 tx=0x7f49b8006b40).stop 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 msgr2=0x7f49c81056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c81056f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f49c40089f0 tx=0x7f49c400b040).stop 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 shutdown_connections 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f49b4031090 0x7f49b4033550 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:08.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 --2- 172.21.15.169:0/3555379128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80f5480 0x7f49c81056f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:08.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 >> 172.21.15.169:0/3555379128 conn(0x7f49c80f0be0 msgr2=0x7f49c80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:08.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 shutdown_connections 2024-01-11T05:06:08.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:08.489+0000 7f49cf7d7700 1 -- 172.21.15.169:0/3555379128 wait complete. 2024-01-11T05:06:08.491 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:08.294504+0000 mgr.smithi067.srbzpc (mgr.14156) 85 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:08.488924+0000 mon.smithi067 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.169:0/3555379128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:09.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:09.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:10.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 -- 172.21.15.169:0/3842149240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 msgr2=0x7f8d700f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:10.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 --2- 172.21.15.169:0/3842149240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d700f69b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8d64004760 tx=0x7f8d64009ad0).stop 2024-01-11T05:06:10.979 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 -- 172.21.15.169:0/3842149240 shutdown_connections 2024-01-11T05:06:10.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 --2- 172.21.15.169:0/3842149240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d700f69b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:10.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 -- 172.21.15.169:0/3842149240 >> 172.21.15.169:0/3842149240 conn(0x7f8d700f1cd0 msgr2=0x7f8d700f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:10.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 -- 172.21.15.169:0/3842149240 shutdown_connections 2024-01-11T05:06:10.980 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.978+0000 7f8d75627700 1 -- 172.21.15.169:0/3842149240 wait complete. 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d75627700 1 Processor -- start 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d75627700 1 -- start start 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d75627700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d701067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d75627700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70106cb0 con 0x7f8d700f6590 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d6effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d701067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d6effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d701067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:53230/0 (socket says 172.21.15.169:53230) 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.979+0000 7f8d6effd700 1 -- 172.21.15.169:0/4278631753 learned_addr learned my addr 172.21.15.169:0/4278631753 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d6effd700 1 -- 172.21.15.169:0/4278631753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d64009d70 con 0x7f8d700f6590 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d6effd700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d701067c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8d64004170 tx=0x7f8d64009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:10.982 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d6400e040 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d70106eb0 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8d64008ef0 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d64014c90 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.980+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d70107340 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.981+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8d640283f0 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.981+0000 7f8d577fe700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8d58031090 0x7f8d58033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.981+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8d64042260 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.981+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d70045bb0 con 0x7f8d700f6590 2024-01-11T05:06:10.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.981+0000 7f8d6e7fc700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8d58031090 0x7f8d58033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:10.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.982+0000 7f8d6e7fc700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8d58031090 0x7f8d58033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8d600096a0 tx=0x7f8d60006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:10.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:10.986+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8d640268f0 con 0x7f8d700f6590 2024-01-11T05:06:11.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.193+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d70107fc0 con 0x7f8d700f6590 2024-01-11T05:06:11.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.193+0000 7f8d577fe700 1 -- 172.21.15.169:0/4278631753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d64026300 con 0x7f8d700f6590 2024-01-11T05:06:11.194 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:11.194 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:11.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8d58031090 msgr2=0x7f8d58033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:11.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8d58031090 0x7f8d58033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8d600096a0 tx=0x7f8d60006b40).stop 2024-01-11T05:06:11.196 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 msgr2=0x7f8d701067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d701067c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8d64004170 tx=0x7f8d64009ad0).stop 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 shutdown_connections 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8d58031090 0x7f8d58033550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 --2- 172.21.15.169:0/4278631753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d700f6590 0x7f8d701067c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 >> 172.21.15.169:0/4278631753 conn(0x7f8d700f1cd0 msgr2=0x7f8d700f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 shutdown_connections 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:11.195+0000 7f8d75627700 1 -- 172.21.15.169:0/4278631753 wait complete. 2024-01-11T05:06:11.197 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:10.294898+0000 mgr.smithi067.srbzpc (mgr.14156) 86 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:11.194755+0000 mon.smithi067 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.169:0/4278631753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:12.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:12.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 -- 172.21.15.169:0/778636225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 msgr2=0x7fbea00f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 --2- 172.21.15.169:0/778636225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea00f6800 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fbe88009fb0 tx=0x7fbe88009b30).stop 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 -- 172.21.15.169:0/778636225 shutdown_connections 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 --2- 172.21.15.169:0/778636225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea00f6800 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 -- 172.21.15.169:0/778636225 >> 172.21.15.169:0/778636225 conn(0x7fbea00f1b40 msgr2=0x7fbea00f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 -- 172.21.15.169:0/778636225 shutdown_connections 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 -- 172.21.15.169:0/778636225 wait complete. 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 Processor -- start 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.670+0000 7fbea4f82700 1 -- start start 2024-01-11T05:06:13.672 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbea4f82700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea0106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:13.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbea4f82700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea0106b10 con 0x7fbea00f63e0 2024-01-11T05:06:13.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbe9f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea0106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:13.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbe9f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea0106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:53256/0 (socket says 172.21.15.169:53256) 2024-01-11T05:06:13.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbe9f7fe700 1 -- 172.21.15.169:0/3953474871 learned_addr learned my addr 172.21.15.169:0/3953474871 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:13.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbe9f7fe700 1 -- 172.21.15.169:0/3953474871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe88005040 con 0x7fbea00f63e0 2024-01-11T05:06:13.673 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbe9f7fe700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea0106620 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fbe880089f0 tx=0x7fbe8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:13.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.671+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe88009320 con 0x7fbea00f63e0 2024-01-11T05:06:13.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.672+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea0106d10 con 0x7fbea00f63e0 2024-01-11T05:06:13.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.672+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe88013ec0 con 0x7fbea00f63e0 2024-01-11T05:06:13.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.672+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe88014d30 con 0x7fbea00f63e0 2024-01-11T05:06:13.674 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.672+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea01071a0 con 0x7fbea00f63e0 2024-01-11T05:06:13.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.672+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbe880095f0 con 0x7fbea00f63e0 2024-01-11T05:06:13.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.672+0000 7fbe97fff700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe8c031090 0x7fbe8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:13.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.673+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbe880424a0 con 0x7fbea00f63e0 2024-01-11T05:06:13.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.673+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea0045bb0 con 0x7fbea00f63e0 2024-01-11T05:06:13.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.673+0000 7fbe9effd700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe8c031090 0x7fbe8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:13.675 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.673+0000 7fbe9effd700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe8c031090 0x7fbe8c033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbe900096a0 tx=0x7fbe90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:13.678 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.677+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbe8801dd70 con 0x7fbea00f63e0 2024-01-11T05:06:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:12.295205+0000 mgr.smithi067.srbzpc (mgr.14156) 87 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.888+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbea00fbfc0 con 0x7fbea00f63e0 2024-01-11T05:06:13.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.888+0000 7fbe97fff700 1 -- 172.21.15.169:0/3953474871 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe8802b020 con 0x7fbea00f63e0 2024-01-11T05:06:13.890 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:13.890 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe8c031090 msgr2=0x7fbe8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe8c031090 0x7fbe8c033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbe900096a0 tx=0x7fbe90006b40).stop 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 msgr2=0x7fbea0106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea0106620 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fbe880089f0 tx=0x7fbe8800b040).stop 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 shutdown_connections 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe8c031090 0x7fbe8c033550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 --2- 172.21.15.169:0/3953474871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea00f63e0 0x7fbea0106620 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:13.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 >> 172.21.15.169:0/3953474871 conn(0x7fbea00f1b40 msgr2=0x7fbea00f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 shutdown_connections 2024-01-11T05:06:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:13.890+0000 7fbea4f82700 1 -- 172.21.15.169:0/3953474871 wait complete. 2024-01-11T05:06:13.892 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:14.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:14 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:13.889696+0000 mon.smithi067 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.169:0/3953474871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:15.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:15.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:15.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:14.295477+0000 mgr.smithi067.srbzpc (mgr.14156) 88 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 -- 172.21.15.169:0/3703784246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 msgr2=0x7f5b940f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 --2- 172.21.15.169:0/3703784246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b940f69b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5b84004760 tx=0x7f5b84009ad0).stop 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 -- 172.21.15.169:0/3703784246 shutdown_connections 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 --2- 172.21.15.169:0/3703784246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b940f69b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 -- 172.21.15.169:0/3703784246 >> 172.21.15.169:0/3703784246 conn(0x7f5b940f1cd0 msgr2=0x7f5b940f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 -- 172.21.15.169:0/3703784246 shutdown_connections 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 -- 172.21.15.169:0/3703784246 wait complete. 2024-01-11T05:06:16.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 Processor -- start 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.407+0000 7f5b9c755700 1 -- start start 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9c755700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b941067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9c755700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b94106cb0 con 0x7f5b940f6590 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9a4f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b941067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9a4f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b941067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:53276/0 (socket says 172.21.15.169:53276) 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9a4f1700 1 -- 172.21.15.169:0/1062737322 learned_addr learned my addr 172.21.15.169:0/1062737322 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9a4f1700 1 -- 172.21.15.169:0/1062737322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84009d70 con 0x7f5b940f6590 2024-01-11T05:06:16.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9a4f1700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b941067c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f5b84004170 tx=0x7f5b84009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b8400e040 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.408+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b94106eb0 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.409+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5b84008ef0 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.409+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b84014c90 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.409+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b94107340 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.409+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5b84014420 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.409+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b94045bb0 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.409+0000 7f5b8affd700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5b80031090 0x7f5b80033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.410+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5b840422b0 con 0x7f5b940f6590 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.410+0000 7f5b99cf0700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5b80031090 0x7f5b80033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:16.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.410+0000 7f5b99cf0700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5b80031090 0x7f5b80033550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:16.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.414+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5b84026300 con 0x7f5b940f6590 2024-01-11T05:06:16.622 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.621+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b941081f0 con 0x7f5b940f6590 2024-01-11T05:06:16.623 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b8affd700 1 -- 172.21.15.169:0/1062737322 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b84012b20 con 0x7f5b940f6590 2024-01-11T05:06:16.623 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:16.623 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5b80031090 msgr2=0x7f5b80033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5b80031090 0x7f5b80033550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5b900096a0 tx=0x7f5b90006b40).stop 2024-01-11T05:06:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 msgr2=0x7f5b941067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b941067c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f5b84004170 tx=0x7f5b84009ad0).stop 2024-01-11T05:06:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 shutdown_connections 2024-01-11T05:06:16.625 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5b80031090 0x7f5b80033550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:16.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 --2- 172.21.15.169:0/1062737322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b940f6590 0x7f5b941067c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:16.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 >> 172.21.15.169:0/1062737322 conn(0x7f5b940f1cd0 msgr2=0x7f5b940f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:16.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 shutdown_connections 2024-01-11T05:06:16.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:16.623+0000 7f5b9c755700 1 -- 172.21.15.169:0/1062737322 wait complete. 2024-01-11T05:06:16.626 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:17.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:16.295675+0000 mgr.smithi067.srbzpc (mgr.14156) 89 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:17.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:16.623128+0000 mon.smithi067 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.169:0/1062737322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:17.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:17.989 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 -- 172.21.15.169:0/4206001365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 msgr2=0x7f75640f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 --2- 172.21.15.169:0/4206001365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75640f8fb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f7554009fb0 tx=0x7f7554009b30).stop 2024-01-11T05:06:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 -- 172.21.15.169:0/4206001365 shutdown_connections 2024-01-11T05:06:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 --2- 172.21.15.169:0/4206001365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75640f8fb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 -- 172.21.15.169:0/4206001365 >> 172.21.15.169:0/4206001365 conn(0x7f75640f0a30 msgr2=0x7f75640f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:19.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 -- 172.21.15.169:0/4206001365 shutdown_connections 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 -- 172.21.15.169:0/4206001365 wait complete. 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 Processor -- start 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.113+0000 7f75690f6700 1 -- start start 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f75690f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75641065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f75690f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564106aa0 con 0x7f7564060360 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f7563fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75641065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f7563fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75641065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:53292/0 (socket says 172.21.15.169:53292) 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f7563fff700 1 -- 172.21.15.169:0/2763293030 learned_addr learned my addr 172.21.15.169:0/2763293030 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f7563fff700 1 -- 172.21.15.169:0/2763293030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554005040 con 0x7f7564060360 2024-01-11T05:06:19.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f7563fff700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75641065b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f7554009f80 tx=0x7f755400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7554009320 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.114+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564106ca0 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.115+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7554013680 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.115+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f755401eda0 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.115+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7564107130 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.115+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f755401e420 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.115+0000 7f7560ff9700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7550031090 0x7f7550033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.116+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7554042380 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.115+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7564045bb0 con 0x7f7564060360 2024-01-11T05:06:19.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.116+0000 7f75637fe700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7550031090 0x7f7550033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:19.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.116+0000 7f75637fe700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7550031090 0x7f7550033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f754c0096a0 tx=0x7f754c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:19.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.121+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f755401e700 con 0x7f7564060360 2024-01-11T05:06:19.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.332+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7564107f30 con 0x7f7564060360 2024-01-11T05:06:19.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.333+0000 7f7560ff9700 1 -- 172.21.15.169:0/2763293030 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f755401cb10 con 0x7f7564060360 2024-01-11T05:06:19.334 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:19.334 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7550031090 msgr2=0x7f7550033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7550031090 0x7f7550033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f754c0096a0 tx=0x7f754c006b40).stop 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 msgr2=0x7f75641065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75641065b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f7554009f80 tx=0x7f755400b040).stop 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 shutdown_connections 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7550031090 0x7f7550033550 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 --2- 172.21.15.169:0/2763293030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7564060360 0x7f75641065b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 >> 172.21.15.169:0/2763293030 conn(0x7f75640f0a30 msgr2=0x7f75640f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 shutdown_connections 2024-01-11T05:06:19.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:19.335+0000 7f75690f6700 1 -- 172.21.15.169:0/2763293030 wait complete. 2024-01-11T05:06:19.337 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:18.295964+0000 mgr.smithi067.srbzpc (mgr.14156) 90 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:19.334369+0000 mon.smithi067 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.169:0/2763293030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:20.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:20.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:20.296206+0000 mgr.smithi067.srbzpc (mgr.14156) 91 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:21.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 -- 172.21.15.169:0/651504879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 msgr2=0x7f88a80f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 --2- 172.21.15.169:0/651504879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a80f6800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f889c009fb0 tx=0x7f889c009b30).stop 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 -- 172.21.15.169:0/651504879 shutdown_connections 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 --2- 172.21.15.169:0/651504879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a80f6800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 -- 172.21.15.169:0/651504879 >> 172.21.15.169:0/651504879 conn(0x7f88a80f1b40 msgr2=0x7f88a80f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 -- 172.21.15.169:0/651504879 shutdown_connections 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.887+0000 7f88adfdb700 1 -- 172.21.15.169:0/651504879 wait complete. 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88adfdb700 1 Processor -- start 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88adfdb700 1 -- start start 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88adfdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a8106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88adfdb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a8106b10 con 0x7f88a80f63e0 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88acfd9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a8106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:21.890 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88acfd9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a8106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39678/0 (socket says 172.21.15.169:39678) 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88acfd9700 1 -- 172.21.15.169:0/971474190 learned_addr learned my addr 172.21.15.169:0/971474190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.888+0000 7f88acfd9700 1 -- 172.21.15.169:0/971474190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c005040 con 0x7f88a80f63e0 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.889+0000 7f88acfd9700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a8106620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f889c0089f0 tx=0x7f889c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.889+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c009320 con 0x7f88a80f63e0 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.889+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a8106d10 con 0x7f88a80f63e0 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.889+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f889c013ec0 con 0x7f88a80f63e0 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.889+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c014d30 con 0x7f88a80f63e0 2024-01-11T05:06:21.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.889+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a81071a0 con 0x7f88a80f63e0 2024-01-11T05:06:21.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.890+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f889c0263a0 con 0x7f88a80f63e0 2024-01-11T05:06:21.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.890+0000 7f88a57fa700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8898031090 0x7f8898033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:21.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.890+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f889c042440 con 0x7f88a80f63e0 2024-01-11T05:06:21.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.890+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a8045bb0 con 0x7f88a80f63e0 2024-01-11T05:06:21.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.890+0000 7f88a7fff700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8898031090 0x7f8898033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:21.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.891+0000 7f88a7fff700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8898031090 0x7f8898033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f88940096a0 tx=0x7f8894006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:21.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:21.895+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f889c0126f0 con 0x7f88a80f63e0 2024-01-11T05:06:22.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.113+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88a80fbfc0 con 0x7f88a80f63e0 2024-01-11T05:06:22.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.113+0000 7f88a57fa700 1 -- 172.21.15.169:0/971474190 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f889c02b020 con 0x7f88a80f63e0 2024-01-11T05:06:22.117 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:22.117 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8898031090 msgr2=0x7f8898033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8898031090 0x7f8898033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f88940096a0 tx=0x7f8894006b40).stop 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 msgr2=0x7f88a8106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a8106620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f889c0089f0 tx=0x7f889c00b040).stop 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 shutdown_connections 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8898031090 0x7f8898033550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 --2- 172.21.15.169:0/971474190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a80f63e0 0x7f88a8106620 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 >> 172.21.15.169:0/971474190 conn(0x7f88a80f1b40 msgr2=0x7f88a80f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 shutdown_connections 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:22.115+0000 7f88adfdb700 1 -- 172.21.15.169:0/971474190 wait complete. 2024-01-11T05:06:22.118 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:22.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:22 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:22.114606+0000 mon.smithi067 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.169:0/971474190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:23.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:23.518 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:22.296536+0000 mgr.smithi067.srbzpc (mgr.14156) 92 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:22.413874+0000 mon.smithi067 (mon.0) 169 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:06:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:22.414555+0000 mon.smithi067 (mon.0) 170 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:06:24.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 -- 172.21.15.169:0/2632966227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 msgr2=0x7fd67c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 --2- 172.21.15.169:0/2632966227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c0f56f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fd678009fb0 tx=0x7fd678009b30).stop 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 -- 172.21.15.169:0/2632966227 shutdown_connections 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 --2- 172.21.15.169:0/2632966227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c0f56f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 -- 172.21.15.169:0/2632966227 >> 172.21.15.169:0/2632966227 conn(0x7fd67c0f0a30 msgr2=0x7fd67c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 -- 172.21.15.169:0/2632966227 shutdown_connections 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.694+0000 7fd682d1c700 1 -- 172.21.15.169:0/2632966227 wait complete. 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd682d1c700 1 Processor -- start 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd682d1c700 1 -- start start 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd682d1c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c1092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:24.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd682d1c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c060360 con 0x7fd67c0f52d0 2024-01-11T05:06:24.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd681d1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c1092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:24.698 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd681d1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c1092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39694/0 (socket says 172.21.15.169:39694) 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.695+0000 7fd681d1a700 1 -- 172.21.15.169:0/4277226375 learned_addr learned my addr 172.21.15.169:0/4277226375 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd681d1a700 1 -- 172.21.15.169:0/4277226375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678005040 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd681d1a700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c1092d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd6780089f0 tx=0x7fd67800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd678009320 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c060560 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd678013ec0 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd678014d30 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.696+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c109a00 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.697+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd6780095f0 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.697+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c045bb0 con 0x7fd67c0f52d0 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.697+0000 7fd6727fc700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd668031090 0x7fd668033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:24.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.697+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd6780423e0 con 0x7fd67c0f52d0 2024-01-11T05:06:24.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.697+0000 7fd681519700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd668031090 0x7fd668033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:24.700 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.698+0000 7fd681519700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd668031090 0x7fd668033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd66c0096a0 tx=0x7fd66c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:24.703 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.703+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd67801d980 con 0x7fd67c0f52d0 2024-01-11T05:06:24.911 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.910+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd67c0fadc0 con 0x7fd67c0f52d0 2024-01-11T05:06:24.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.912+0000 7fd6727fc700 1 -- 172.21.15.169:0/4277226375 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd67801d380 con 0x7fd67c0f52d0 2024-01-11T05:06:24.912 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:24.912 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.912+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd668031090 msgr2=0x7fd668033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.912+0000 7fd682d1c700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd668031090 0x7fd668033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd66c0096a0 tx=0x7fd66c006b40).stop 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.912+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 msgr2=0x7fd67c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.912+0000 7fd682d1c700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c1092d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd6780089f0 tx=0x7fd67800b040).stop 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.913+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 shutdown_connections 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.913+0000 7fd682d1c700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd668031090 0x7fd668033550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.913+0000 7fd682d1c700 1 --2- 172.21.15.169:0/4277226375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c0f52d0 0x7fd67c1092d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.913+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 >> 172.21.15.169:0/4277226375 conn(0x7fd67c0f0a30 msgr2=0x7fd67c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.913+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 shutdown_connections 2024-01-11T05:06:24.914 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:24.913+0000 7fd682d1c700 1 -- 172.21.15.169:0/4277226375 wait complete. 2024-01-11T05:06:24.915 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:24.296977+0000 mgr.smithi067.srbzpc (mgr.14156) 93 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:24.912328+0000 mon.smithi067 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.169:0/4277226375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:26.284 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:26.284 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 -- 172.21.15.169:0/584789699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 msgr2=0x7f80c80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 --2- 172.21.15.169:0/584789699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c80f58a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f80b8009fb0 tx=0x7f80b8009b30).stop 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 -- 172.21.15.169:0/584789699 shutdown_connections 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 --2- 172.21.15.169:0/584789699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c80f58a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 -- 172.21.15.169:0/584789699 >> 172.21.15.169:0/584789699 conn(0x7f80c80f0be0 msgr2=0x7f80c80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 -- 172.21.15.169:0/584789699 shutdown_connections 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.402+0000 7f80cfb02700 1 -- 172.21.15.169:0/584789699 wait complete. 2024-01-11T05:06:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cfb02700 1 Processor -- start 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cfb02700 1 -- start start 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cfb02700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c81056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cfb02700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c8105be0 con 0x7f80c80f5480 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cd89e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c81056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cd89e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c81056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39714/0 (socket says 172.21.15.169:39714) 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.403+0000 7f80cd89e700 1 -- 172.21.15.169:0/2591412666 learned_addr learned my addr 172.21.15.169:0/2591412666 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80cd89e700 1 -- 172.21.15.169:0/2591412666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b8005040 con 0x7f80c80f5480 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80cd89e700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c81056f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f80b80089f0 tx=0x7f80b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b8009320 con 0x7f80c80f5480 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c8105de0 con 0x7f80c80f5480 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f80b8013ec0 con 0x7f80c80f5480 2024-01-11T05:06:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b8014d30 con 0x7f80c80f5480 2024-01-11T05:06:27.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.404+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c8106270 con 0x7f80c80f5480 2024-01-11T05:06:27.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.405+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f80b80095f0 con 0x7f80c80f5480 2024-01-11T05:06:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.405+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c8045bb0 con 0x7f80c80f5480 2024-01-11T05:06:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.405+0000 7f80be7fc700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80b40310e0 0x7f80b40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.405+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f80b80423a0 con 0x7f80c80f5480 2024-01-11T05:06:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.405+0000 7f80cd09d700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80b40310e0 0x7f80b40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.405+0000 7f80cd09d700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80b40310e0 0x7f80b40335a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f80c40096a0 tx=0x7f80c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.410+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f80b801d7f0 con 0x7f80c80f5480 2024-01-11T05:06:27.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.618+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80c80faf70 con 0x7f80c80f5480 2024-01-11T05:06:27.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.619+0000 7f80be7fc700 1 -- 172.21.15.169:0/2591412666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80b801d7f0 con 0x7f80c80f5480 2024-01-11T05:06:27.619 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:27.619 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:27.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80b40310e0 msgr2=0x7f80b40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80b40310e0 0x7f80b40335a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f80c40096a0 tx=0x7f80c4006b40).stop 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 msgr2=0x7f80c81056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c81056f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f80b80089f0 tx=0x7f80b800b040).stop 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 shutdown_connections 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80b40310e0 0x7f80b40335a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 --2- 172.21.15.169:0/2591412666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c80f5480 0x7f80c81056f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 >> 172.21.15.169:0/2591412666 conn(0x7f80c80f0be0 msgr2=0x7f80c80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 shutdown_connections 2024-01-11T05:06:27.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:27.621+0000 7f80cfb02700 1 -- 172.21.15.169:0/2591412666 wait complete. 2024-01-11T05:06:27.622 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:26.297351+0000 mgr.smithi067.srbzpc (mgr.14156) 94 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:28.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:28 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:27.619351+0000 mon.smithi067 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.169:0/2591412666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:28.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:29.000 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:28.297735+0000 mgr.smithi067.srbzpc (mgr.14156) 95 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:30.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 -- 172.21.15.169:0/1394466236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 msgr2=0x7f9a180f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:30.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 --2- 172.21.15.169:0/1394466236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a180f6800 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f9a08009fb0 tx=0x7f9a08009b30).stop 2024-01-11T05:06:30.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 -- 172.21.15.169:0/1394466236 shutdown_connections 2024-01-11T05:06:30.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 --2- 172.21.15.169:0/1394466236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a180f6800 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:30.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 -- 172.21.15.169:0/1394466236 >> 172.21.15.169:0/1394466236 conn(0x7f9a180f1b40 msgr2=0x7f9a180f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:30.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 -- 172.21.15.169:0/1394466236 shutdown_connections 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 -- 172.21.15.169:0/1394466236 wait complete. 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 Processor -- start 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.130+0000 7f9a1759e700 1 -- start start 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a1806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1759e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a18069f00 con 0x7f9a180f63e0 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a1806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a1806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:34188/0 (socket says 172.21.15.169:34188) 2024-01-11T05:06:30.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1659c700 1 -- 172.21.15.169:0/2119184158 learned_addr learned my addr 172.21.15.169:0/2119184158 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1659c700 1 -- 172.21.15.169:0/2119184158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08005040 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1659c700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a1806ba80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f9a080089f0 tx=0x7f9a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a08009320 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1806a0a0 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9a08013ec0 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a08014d30 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.131+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1806a530 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.132+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9a080095f0 con 0x7f9a180f63e0 2024-01-11T05:06:30.136 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.132+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a180591e0 con 0x7f9a180f63e0 2024-01-11T05:06:30.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.132+0000 7f9a06ffd700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9a00031090 0x7f9a00033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:30.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.132+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9a08042420 con 0x7f9a180f63e0 2024-01-11T05:06:30.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.133+0000 7f9a15d9b700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9a00031090 0x7f9a00033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:30.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.133+0000 7f9a15d9b700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9a00031090 0x7f9a00033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f9a0c0096a0 tx=0x7f9a0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:30.138 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.137+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9a0801d7f0 con 0x7f9a180f63e0 2024-01-11T05:06:30.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.346+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a180fbed0 con 0x7f9a180f63e0 2024-01-11T05:06:30.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.346+0000 7f9a06ffd700 1 -- 172.21.15.169:0/2119184158 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a0800fe40 con 0x7f9a180f63e0 2024-01-11T05:06:30.347 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:30.347 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9a00031090 msgr2=0x7f9a00033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9a00031090 0x7f9a00033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f9a0c0096a0 tx=0x7f9a0c006b40).stop 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 msgr2=0x7f9a1806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a1806ba80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f9a080089f0 tx=0x7f9a0800b040).stop 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 shutdown_connections 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9a00031090 0x7f9a00033550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 --2- 172.21.15.169:0/2119184158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a180f63e0 0x7f9a1806ba80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 >> 172.21.15.169:0/2119184158 conn(0x7f9a180f1b40 msgr2=0x7f9a180f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:30.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 shutdown_connections 2024-01-11T05:06:30.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:30.348+0000 7f9a1759e700 1 -- 172.21.15.169:0/2119184158 wait complete. 2024-01-11T05:06:30.349 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:30.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:30 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:30.346872+0000 mon.smithi067 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.169:0/2119184158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:31.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:31.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:30.298049+0000 mgr.smithi067.srbzpc (mgr.14156) 96 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:32.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 -- 172.21.15.169:0/2796076897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 msgr2=0x7fc8040f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:32.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 --2- 172.21.15.169:0/2796076897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc8040f91c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fc7f8009fb0 tx=0x7fc7f8009b30).stop 2024-01-11T05:06:32.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 -- 172.21.15.169:0/2796076897 shutdown_connections 2024-01-11T05:06:32.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 --2- 172.21.15.169:0/2796076897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc8040f91c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 -- 172.21.15.169:0/2796076897 >> 172.21.15.169:0/2796076897 conn(0x7fc8040f0be0 msgr2=0x7fc8040f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 -- 172.21.15.169:0/2796076897 shutdown_connections 2024-01-11T05:06:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.860+0000 7fc8092f6700 1 -- 172.21.15.169:0/2796076897 wait complete. 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc8092f6700 1 Processor -- start 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc8092f6700 1 -- start start 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc8092f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc804106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc8092f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804106c70 con 0x7fc804060300 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc802ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc804106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc802ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc804106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:34196/0 (socket says 172.21.15.169:34196) 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.861+0000 7fc802ffd700 1 -- 172.21.15.169:0/800920717 learned_addr learned my addr 172.21.15.169:0/800920717 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc802ffd700 1 -- 172.21.15.169:0/800920717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8005040 con 0x7fc804060300 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc802ffd700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc804106780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc7f8000c00 tx=0x7fc7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f80095f0 con 0x7fc804060300 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc7f8007e50 con 0x7fc804060300 2024-01-11T05:06:32.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc804106e70 con 0x7fc804060300 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8014e70 con 0x7fc804060300 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.862+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc804107300 con 0x7fc804060300 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.863+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc7f8025070 con 0x7fc804060300 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.863+0000 7fc7eb7fe700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7ec031090 0x7fc7ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.863+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc804045bb0 con 0x7fc804060300 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.863+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc7f8041d10 con 0x7fc804060300 2024-01-11T05:06:32.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.863+0000 7fc8027fc700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7ec031090 0x7fc7ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:32.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.863+0000 7fc8027fc700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7ec031090 0x7fc7ec033550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc7f40096a0 tx=0x7fc7f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:32.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:32.868+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc7f800f4e0 con 0x7fc804060300 2024-01-11T05:06:33.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.075+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc804108180 con 0x7fc804060300 2024-01-11T05:06:33.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.076+0000 7fc7eb7fe700 1 -- 172.21.15.169:0/800920717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7f801ea30 con 0x7fc804060300 2024-01-11T05:06:33.076 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:33.076 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:33.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7ec031090 msgr2=0x7fc7ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:33.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7ec031090 0x7fc7ec033550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc7f40096a0 tx=0x7fc7f4006b40).stop 2024-01-11T05:06:33.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 msgr2=0x7fc804106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:33.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc804106780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc7f8000c00 tx=0x7fc7f800b040).stop 2024-01-11T05:06:33.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 shutdown_connections 2024-01-11T05:06:33.078 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7ec031090 0x7fc7ec033550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:33.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 --2- 172.21.15.169:0/800920717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc804060300 0x7fc804106780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:33.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 >> 172.21.15.169:0/800920717 conn(0x7fc8040f0be0 msgr2=0x7fc8040f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:33.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 shutdown_connections 2024-01-11T05:06:33.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:33.078+0000 7fc8092f6700 1 -- 172.21.15.169:0/800920717 wait complete. 2024-01-11T05:06:33.079 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:32.298355+0000 mgr.smithi067.srbzpc (mgr.14156) 97 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:33.076497+0000 mon.smithi067 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.169:0/800920717' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:34.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:34.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:34.298663+0000 mgr.smithi067.srbzpc (mgr.14156) 98 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 -- 172.21.15.169:0/3311289989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 msgr2=0x7f3fdc0f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 --2- 172.21.15.169:0/3311289989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc0f6800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f3fcc009fb0 tx=0x7f3fcc009b30).stop 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 -- 172.21.15.169:0/3311289989 shutdown_connections 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 --2- 172.21.15.169:0/3311289989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc0f6800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 -- 172.21.15.169:0/3311289989 >> 172.21.15.169:0/3311289989 conn(0x7f3fdc0f1b40 msgr2=0x7f3fdc0f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 -- 172.21.15.169:0/3311289989 shutdown_connections 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.544+0000 7f3fe3000700 1 -- 172.21.15.169:0/3311289989 wait complete. 2024-01-11T05:06:35.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe3000700 1 Processor -- start 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe3000700 1 -- start start 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe3000700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe3000700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc106b10 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe1ffe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe1ffe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:34210/0 (socket says 172.21.15.169:34210) 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe1ffe700 1 -- 172.21.15.169:0/3715781893 learned_addr learned my addr 172.21.15.169:0/3715781893 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.545+0000 7f3fe1ffe700 1 -- 172.21.15.169:0/3715781893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc005040 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.546+0000 7f3fe1ffe700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc106620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f3fcc0089f0 tx=0x7f3fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.546+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc009320 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.546+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc106d10 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.546+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3fcc013ec0 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.546+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fcc014d30 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.546+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc1071a0 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.547+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3fcc0095f0 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.547+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc045bb0 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.547+0000 7f3fd27fc700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc8031090 0x7f3fc8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:35.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.547+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3fcc0424a0 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.547+0000 7f3fe17fd700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc8031090 0x7f3fc8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:35.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.547+0000 7f3fe17fd700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc8031090 0x7f3fc8033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3fd80096a0 tx=0x7f3fd8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:35.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.552+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3fcc01d980 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.761+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fdc108050 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.762 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.762+0000 7f3fd27fc700 1 -- 172.21.15.169:0/3715781893 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fcc01d380 con 0x7f3fdc0f63e0 2024-01-11T05:06:35.763 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:35.763 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:35.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc8031090 msgr2=0x7f3fc8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:35.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc8031090 0x7f3fc8033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3fd80096a0 tx=0x7f3fd8006b40).stop 2024-01-11T05:06:35.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 msgr2=0x7f3fdc106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:35.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc106620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f3fcc0089f0 tx=0x7f3fcc00b040).stop 2024-01-11T05:06:35.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 shutdown_connections 2024-01-11T05:06:35.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f3fc8031090 0x7f3fc8033550 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:35.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 --2- 172.21.15.169:0/3715781893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fdc0f63e0 0x7f3fdc106620 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:35.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 >> 172.21.15.169:0/3715781893 conn(0x7f3fdc0f1b40 msgr2=0x7f3fdc0f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:35.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 shutdown_connections 2024-01-11T05:06:35.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:35.763+0000 7f3fe3000700 1 -- 172.21.15.169:0/3715781893 wait complete. 2024-01-11T05:06:35.765 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:36.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:35.763011+0000 mon.smithi067 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.169:0/3715781893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:37.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:37.141 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:37.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:36.298971+0000 mgr.smithi067.srbzpc (mgr.14156) 99 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:38.252 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 -- 172.21.15.169:0/4077372337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 msgr2=0x7fa6900f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 --2- 172.21.15.169:0/4077372337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa6900f6800 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fa680009fb0 tx=0x7fa680009b30).stop 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 -- 172.21.15.169:0/4077372337 shutdown_connections 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 --2- 172.21.15.169:0/4077372337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa6900f6800 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 -- 172.21.15.169:0/4077372337 >> 172.21.15.169:0/4077372337 conn(0x7fa6900f1b40 msgr2=0x7fa6900f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 -- 172.21.15.169:0/4077372337 shutdown_connections 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 -- 172.21.15.169:0/4077372337 wait complete. 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.250+0000 7fa694f7d700 1 Processor -- start 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa694f7d700 1 -- start start 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa694f7d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa690106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa694f7d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa690106b50 con 0x7fa6900f63e0 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa68f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa690106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa68f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa690106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:34230/0 (socket says 172.21.15.169:34230) 2024-01-11T05:06:38.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa68f7fe700 1 -- 172.21.15.169:0/303191696 learned_addr learned my addr 172.21.15.169:0/303191696 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa68f7fe700 1 -- 172.21.15.169:0/303191696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa680005040 con 0x7fa6900f63e0 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.251+0000 7fa68f7fe700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa690106660 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa6800089f0 tx=0x7fa68000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.252+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa680009320 con 0x7fa6900f63e0 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.252+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa690106d50 con 0x7fa6900f63e0 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.252+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa680013ec0 con 0x7fa6900f63e0 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.252+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa680014d30 con 0x7fa6900f63e0 2024-01-11T05:06:38.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.252+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6901071e0 con 0x7fa6900f63e0 2024-01-11T05:06:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.253+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa680014420 con 0x7fa6900f63e0 2024-01-11T05:06:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.253+0000 7fa677fff700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa678031090 0x7fa678033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.253+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa680042400 con 0x7fa6900f63e0 2024-01-11T05:06:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.253+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa690045bb0 con 0x7fa6900f63e0 2024-01-11T05:06:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.253+0000 7fa68effd700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa678031090 0x7fa678033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:38.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.254+0000 7fa68effd700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa678031090 0x7fa678033550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa6840096a0 tx=0x7fa684006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:38.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.258+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa680014750 con 0x7fa6900f63e0 2024-01-11T05:06:38.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.468+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa690107f00 con 0x7fa6900f63e0 2024-01-11T05:06:38.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.469+0000 7fa677fff700 1 -- 172.21.15.169:0/303191696 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa68001d380 con 0x7fa6900f63e0 2024-01-11T05:06:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:38.470 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa678031090 msgr2=0x7fa678033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa678031090 0x7fa678033550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa6840096a0 tx=0x7fa684006b40).stop 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 msgr2=0x7fa690106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa690106660 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa6800089f0 tx=0x7fa68000b040).stop 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 shutdown_connections 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa678031090 0x7fa678033550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 --2- 172.21.15.169:0/303191696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6900f63e0 0x7fa690106660 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 >> 172.21.15.169:0/303191696 conn(0x7fa6900f1b40 msgr2=0x7fa6900f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 shutdown_connections 2024-01-11T05:06:38.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:38.471+0000 7fa694f7d700 1 -- 172.21.15.169:0/303191696 wait complete. 2024-01-11T05:06:38.473 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:38.299288+0000 mgr.smithi067.srbzpc (mgr.14156) 100 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:38.470456+0000 mon.smithi067 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.169:0/303191696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:39.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:39.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:40.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.957+0000 7fbab9f5c700 1 -- 172.21.15.169:0/3937136152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 msgr2=0x7fbab40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:40.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.957+0000 7fbab9f5c700 1 --2- 172.21.15.169:0/3937136152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab40f56f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fbaa8009fb0 tx=0x7fbaa8009b30).stop 2024-01-11T05:06:40.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 -- 172.21.15.169:0/3937136152 shutdown_connections 2024-01-11T05:06:40.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 --2- 172.21.15.169:0/3937136152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab40f56f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:40.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 -- 172.21.15.169:0/3937136152 >> 172.21.15.169:0/3937136152 conn(0x7fbab40f0a30 msgr2=0x7fbab40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:40.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 -- 172.21.15.169:0/3937136152 shutdown_connections 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 -- 172.21.15.169:0/3937136152 wait complete. 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 Processor -- start 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 -- start start 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab4106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.958+0000 7fbab9f5c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4106b30 con 0x7fbab40f52d0 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab8f5a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab4106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab8f5a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab4106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36604/0 (socket says 172.21.15.169:36604) 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab8f5a700 1 -- 172.21.15.169:0/743061064 learned_addr learned my addr 172.21.15.169:0/743061064 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab8f5a700 1 -- 172.21.15.169:0/743061064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa8005040 con 0x7fbab40f52d0 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab8f5a700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab4106640 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fbaa80089f0 tx=0x7fbaa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa8009320 con 0x7fbab40f52d0 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab4106d30 con 0x7fbab40f52d0 2024-01-11T05:06:40.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab41071c0 con 0x7fbab40f52d0 2024-01-11T05:06:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbaa8013ec0 con 0x7fbab40f52d0 2024-01-11T05:06:40.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.959+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaa8014d00 con 0x7fbab40f52d0 2024-01-11T05:06:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.960+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbaa8014420 con 0x7fbab40f52d0 2024-01-11T05:06:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.960+0000 7fbab17fa700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbaa4031090 0x7fbaa4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.960+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab4045bb0 con 0x7fbab40f52d0 2024-01-11T05:06:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.960+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbaa80423e0 con 0x7fbab40f52d0 2024-01-11T05:06:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.961+0000 7fbab3fff700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbaa4031090 0x7fbaa4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:40.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.961+0000 7fbab3fff700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbaa4031090 0x7fbaa4033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbaa00096a0 tx=0x7fbaa0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:40.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:40.965+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbaa8012d00 con 0x7fbab40f52d0 2024-01-11T05:06:41.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.171+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbab4108070 con 0x7fbab40f52d0 2024-01-11T05:06:41.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.172+0000 7fbab17fa700 1 -- 172.21.15.169:0/743061064 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbaa8012b20 con 0x7fbab40f52d0 2024-01-11T05:06:41.173 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:41.173 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:41.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.173+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbaa4031090 msgr2=0x7fbaa4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:41.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.173+0000 7fbab9f5c700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbaa4031090 0x7fbaa4033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbaa00096a0 tx=0x7fbaa0006b40).stop 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.173+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 msgr2=0x7fbab4106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.173+0000 7fbab9f5c700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab4106640 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fbaa80089f0 tx=0x7fbaa800b040).stop 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.173+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 shutdown_connections 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.174+0000 7fbab9f5c700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbaa4031090 0x7fbaa4033550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.174+0000 7fbab9f5c700 1 --2- 172.21.15.169:0/743061064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab40f52d0 0x7fbab4106640 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.174+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 >> 172.21.15.169:0/743061064 conn(0x7fbab40f0a30 msgr2=0x7fbab40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.174+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 shutdown_connections 2024-01-11T05:06:41.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:41.174+0000 7fbab9f5c700 1 -- 172.21.15.169:0/743061064 wait complete. 2024-01-11T05:06:41.176 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:40.299580+0000 mgr.smithi067.srbzpc (mgr.14156) 101 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:41.173090+0000 mon.smithi067 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.169:0/743061064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:42.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:42.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:43.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.661+0000 7f574810f700 1 -- 172.21.15.169:0/3631955897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 msgr2=0x7f57400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:43.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 --2- 172.21.15.169:0/3631955897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f57400f58a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5730009fb0 tx=0x7f5730009b30).stop 2024-01-11T05:06:43.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 -- 172.21.15.169:0/3631955897 shutdown_connections 2024-01-11T05:06:43.663 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 --2- 172.21.15.169:0/3631955897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f57400f58a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 -- 172.21.15.169:0/3631955897 >> 172.21.15.169:0/3631955897 conn(0x7f57400f0be0 msgr2=0x7f57400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 -- 172.21.15.169:0/3631955897 shutdown_connections 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 -- 172.21.15.169:0/3631955897 wait complete. 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 Processor -- start 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.662+0000 7f574810f700 1 -- start start 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f574810f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f5740109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:43.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f574810f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740060300 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f5745eab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f5740109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f5745eab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f5740109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36622/0 (socket says 172.21.15.169:36622) 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f5745eab700 1 -- 172.21.15.169:0/943126232 learned_addr learned my addr 172.21.15.169:0/943126232 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f5745eab700 1 -- 172.21.15.169:0/943126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730005040 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.663+0000 7f5745eab700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f5740109510 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f5730000c00 tx=0x7f573000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.664+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57300095f0 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.664+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5730007e50 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.664+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5740060500 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.664+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5730014e70 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.664+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5740109bc0 con 0x7f57400f5480 2024-01-11T05:06:43.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.665+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5730025070 con 0x7f57400f5480 2024-01-11T05:06:43.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.665+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57400591e0 con 0x7f57400f5480 2024-01-11T05:06:43.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.665+0000 7f57367fc700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f572c0310e0 0x7f572c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:43.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.665+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5730041d20 con 0x7f57400f5480 2024-01-11T05:06:43.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.665+0000 7f57456aa700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f572c0310e0 0x7f572c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:43.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.665+0000 7f57456aa700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f572c0310e0 0x7f572c0335a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:43.670 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.670+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f57300079c0 con 0x7f57400f5480 2024-01-11T05:06:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:42.299917+0000 mgr.smithi067.srbzpc (mgr.14156) 102 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:43.877 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.876+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57400faf70 con 0x7f57400f5480 2024-01-11T05:06:43.878 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.877+0000 7f57367fc700 1 -- 172.21.15.169:0/943126232 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f573000f300 con 0x7f57400f5480 2024-01-11T05:06:43.878 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:43.878 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f572c0310e0 msgr2=0x7f572c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f572c0310e0 0x7f572c0335a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).stop 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 msgr2=0x7f5740109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f5740109510 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f5730000c00 tx=0x7f573000b040).stop 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 shutdown_connections 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f572c0310e0 0x7f572c0335a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 --2- 172.21.15.169:0/943126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57400f5480 0x7f5740109510 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 >> 172.21.15.169:0/943126232 conn(0x7f57400f0be0 msgr2=0x7f57400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 shutdown_connections 2024-01-11T05:06:43.880 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:43.879+0000 7f574810f700 1 -- 172.21.15.169:0/943126232 wait complete. 2024-01-11T05:06:43.881 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:44.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:44 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:43.878570+0000 mon.smithi067 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.169:0/943126232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:45.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:45.262 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:45.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:44.300268+0000 mgr.smithi067.srbzpc (mgr.14156) 103 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 -- 172.21.15.169:0/576705622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 msgr2=0x7f25100f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 --2- 172.21.15.169:0/576705622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f25100f56f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2500009fb0 tx=0x7f2500009b30).stop 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 -- 172.21.15.169:0/576705622 shutdown_connections 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 --2- 172.21.15.169:0/576705622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f25100f56f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 -- 172.21.15.169:0/576705622 >> 172.21.15.169:0/576705622 conn(0x7f25100f0a30 msgr2=0x7f25100f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 -- 172.21.15.169:0/576705622 shutdown_connections 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 -- 172.21.15.169:0/576705622 wait complete. 2024-01-11T05:06:46.443 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 Processor -- start 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.441+0000 7f2514a1d700 1 -- start start 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f2514a1d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f2510106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f2514a1d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2510106af0 con 0x7f25100f52d0 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f250f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f2510106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f250f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f2510106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36638/0 (socket says 172.21.15.169:36638) 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f250f7fe700 1 -- 172.21.15.169:0/2219982253 learned_addr learned my addr 172.21.15.169:0/2219982253 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f250f7fe700 1 -- 172.21.15.169:0/2219982253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500005040 con 0x7f25100f52d0 2024-01-11T05:06:46.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f250f7fe700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f2510106600 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f25000089f0 tx=0x7f250000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:46.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.442+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2500009320 con 0x7f25100f52d0 2024-01-11T05:06:46.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.443+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2500013ec0 con 0x7f25100f52d0 2024-01-11T05:06:46.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.443+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2510106cf0 con 0x7f25100f52d0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.443+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2500014d30 con 0x7f25100f52d0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.443+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2510107180 con 0x7f25100f52d0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.444+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f25000095f0 con 0x7f25100f52d0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.444+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2510045bb0 con 0x7f25100f52d0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.444+0000 7f24f7fff700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24f8031090 0x7f24f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.444+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f25000423e0 con 0x7f25100f52d0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.444+0000 7f250effd700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24f8031090 0x7f24f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:46.446 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.444+0000 7f250effd700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24f8031090 0x7f24f8033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f25040096a0 tx=0x7f2504006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:46.450 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.449+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f250001d7f0 con 0x7f25100f52d0 2024-01-11T05:06:46.657 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.656+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2510108030 con 0x7f25100f52d0 2024-01-11T05:06:46.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.657+0000 7f24f7fff700 1 -- 172.21.15.169:0/2219982253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f250001d7f0 con 0x7f25100f52d0 2024-01-11T05:06:46.658 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:46.658 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24f8031090 msgr2=0x7f24f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24f8031090 0x7f24f8033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f25040096a0 tx=0x7f2504006b40).stop 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 msgr2=0x7f2510106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f2510106600 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f25000089f0 tx=0x7f250000b040).stop 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 shutdown_connections 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24f8031090 0x7f24f8033550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 --2- 172.21.15.169:0/2219982253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25100f52d0 0x7f2510106600 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:46.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 >> 172.21.15.169:0/2219982253 conn(0x7f25100f0a30 msgr2=0x7f251005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:46.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 shutdown_connections 2024-01-11T05:06:46.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:46.659+0000 7f2514a1d700 1 -- 172.21.15.169:0/2219982253 wait complete. 2024-01-11T05:06:46.661 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:46.300497+0000 mgr.smithi067.srbzpc (mgr.14156) 104 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:46.658478+0000 mon.smithi067 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.169:0/2219982253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:48.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:48.028 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:49.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.166+0000 7f65ecd0f700 1 -- 172.21.15.169:0/2873286562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 msgr2=0x7f65e80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:49.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.166+0000 7f65ecd0f700 1 --2- 172.21.15.169:0/2873286562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e80f69b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f65d8009fb0 tx=0x7f65d8009b30).stop 2024-01-11T05:06:49.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.166+0000 7f65ecd0f700 1 -- 172.21.15.169:0/2873286562 shutdown_connections 2024-01-11T05:06:49.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.166+0000 7f65ecd0f700 1 --2- 172.21.15.169:0/2873286562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e80f69b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:49.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.166+0000 7f65ecd0f700 1 -- 172.21.15.169:0/2873286562 >> 172.21.15.169:0/2873286562 conn(0x7f65e80f1cf0 msgr2=0x7f65e80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:49.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65ecd0f700 1 -- 172.21.15.169:0/2873286562 shutdown_connections 2024-01-11T05:06:49.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65ecd0f700 1 -- 172.21.15.169:0/2873286562 wait complete. 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65ecd0f700 1 Processor -- start 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65ecd0f700 1 -- start start 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65ecd0f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65ecd0f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8069f00 con 0x7f65e80f6590 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65e659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65e659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36646/0 (socket says 172.21.15.169:36646) 2024-01-11T05:06:49.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.167+0000 7f65e659c700 1 -- 172.21.15.169:0/1943552992 learned_addr learned my addr 172.21.15.169:0/1943552992 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65e659c700 1 -- 172.21.15.169:0/1943552992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d8005040 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65e659c700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e806ba80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f65d8000c00 tx=0x7f65d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65d80095f0 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e806a0a0 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f65d8007e50 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65d8014e70 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.168+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e806a530 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.169+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f65d80146b0 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.169+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e80591e0 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.169+0000 7f65d6ffd700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f65d0031090 0x7f65d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.169+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f65d8041d10 con 0x7f65e80f6590 2024-01-11T05:06:49.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.169+0000 7f65e5d9b700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f65d0031090 0x7f65d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:49.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.170+0000 7f65e5d9b700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f65d0031090 0x7f65d0033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f65dc0096a0 tx=0x7f65dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:49.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.174+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f65d801b4a0 con 0x7f65e80f6590 2024-01-11T05:06:49.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.381+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65e8045bb0 con 0x7f65e80f6590 2024-01-11T05:06:49.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.382+0000 7f65d6ffd700 1 -- 172.21.15.169:0/1943552992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65d801b2c0 con 0x7f65e80f6590 2024-01-11T05:06:49.383 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:49.383 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f65d0031090 msgr2=0x7f65d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f65d0031090 0x7f65d0033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f65dc0096a0 tx=0x7f65dc006b40).stop 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 msgr2=0x7f65e806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e806ba80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f65d8000c00 tx=0x7f65d800b040).stop 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 shutdown_connections 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f65d0031090 0x7f65d0033550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 --2- 172.21.15.169:0/1943552992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e80f6590 0x7f65e806ba80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 >> 172.21.15.169:0/1943552992 conn(0x7f65e80f1cf0 msgr2=0x7f65e80f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 shutdown_connections 2024-01-11T05:06:49.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:49.384+0000 7f65ecd0f700 1 -- 172.21.15.169:0/1943552992 wait complete. 2024-01-11T05:06:49.386 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:48.300841+0000 mgr.smithi067.srbzpc (mgr.14156) 105 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:50.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:49.383111+0000 mon.smithi067 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.169:0/1943552992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:50.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:50.765 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:50.301160+0000 mgr.smithi067.srbzpc (mgr.14156) 106 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:51.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 -- 172.21.15.169:0/2575965731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 msgr2=0x7f643c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:51.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 --2- 172.21.15.169:0/2575965731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c0f69b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f642c009fb0 tx=0x7f642c009b30).stop 2024-01-11T05:06:51.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 -- 172.21.15.169:0/2575965731 shutdown_connections 2024-01-11T05:06:51.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 --2- 172.21.15.169:0/2575965731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c0f69b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:51.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 -- 172.21.15.169:0/2575965731 >> 172.21.15.169:0/2575965731 conn(0x7f643c0f1cf0 msgr2=0x7f643c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:51.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 -- 172.21.15.169:0/2575965731 shutdown_connections 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 -- 172.21.15.169:0/2575965731 wait complete. 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.884+0000 7f644165c700 1 Processor -- start 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f644165c700 1 -- start start 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f644165c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f644165c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c106c90 con 0x7f643c0f6590 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f643affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f643affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36436/0 (socket says 172.21.15.169:36436) 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f643affd700 1 -- 172.21.15.169:0/3488391016 learned_addr learned my addr 172.21.15.169:0/3488391016 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:51.887 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.885+0000 7f643affd700 1 -- 172.21.15.169:0/3488391016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c005040 con 0x7f643c0f6590 2024-01-11T05:06:51.888 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.886+0000 7f643affd700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c1067a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f642c000c00 tx=0x7f642c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.886+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f642c0095f0 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.886+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f642c007e50 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.886+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c106e90 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.886+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f642c014e70 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.886+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c107320 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.887+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f642c025070 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.887+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c045bb0 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.887+0000 7f64237fe700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6424031090 0x7f6424033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.887+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f642c041d40 con 0x7f643c0f6590 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.887+0000 7f643a7fc700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6424031090 0x7f6424033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:51.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.887+0000 7f643a7fc700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6424031090 0x7f6424033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f64300096a0 tx=0x7f6430006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:51.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:51.891+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f642c00f300 con 0x7f643c0f6590 2024-01-11T05:06:52.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.102+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f643c108130 con 0x7f643c0f6590 2024-01-11T05:06:52.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.103+0000 7f64237fe700 1 -- 172.21.15.169:0/3488391016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f642c00f300 con 0x7f643c0f6590 2024-01-11T05:06:52.104 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:52.104 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:52.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6424031090 msgr2=0x7f6424033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:52.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6424031090 0x7f6424033550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f64300096a0 tx=0x7f6430006b40).stop 2024-01-11T05:06:52.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 msgr2=0x7f643c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:52.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c1067a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f642c000c00 tx=0x7f642c00b040).stop 2024-01-11T05:06:52.105 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 shutdown_connections 2024-01-11T05:06:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6424031090 0x7f6424033550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 --2- 172.21.15.169:0/3488391016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643c0f6590 0x7f643c1067a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 >> 172.21.15.169:0/3488391016 conn(0x7f643c0f1cf0 msgr2=0x7f643c0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 shutdown_connections 2024-01-11T05:06:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:52.104+0000 7f644165c700 1 -- 172.21.15.169:0/3488391016 wait complete. 2024-01-11T05:06:52.106 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:52.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:52.104036+0000 mon.smithi067 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.169:0/3488391016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:53.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:53.489 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:53.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:52.301516+0000 mgr.smithi067.srbzpc (mgr.14156) 107 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:54.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 -- 172.21.15.169:0/32955184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 msgr2=0x7f72d80f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 --2- 172.21.15.169:0/32955184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f8fc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f72d4009fb0 tx=0x7f72d4009b30).stop 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 -- 172.21.15.169:0/32955184 shutdown_connections 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 --2- 172.21.15.169:0/32955184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f8fc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 -- 172.21.15.169:0/32955184 >> 172.21.15.169:0/32955184 conn(0x7f72d80f0a40 msgr2=0x7f72d80f2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 -- 172.21.15.169:0/32955184 shutdown_connections 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 -- 172.21.15.169:0/32955184 wait complete. 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 Processor -- start 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.616+0000 7f72df107700 1 -- start start 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72df107700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72df107700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80f4ec0 con 0x7f72d8060360 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72de105700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72de105700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36450/0 (socket says 172.21.15.169:36450) 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72de105700 1 -- 172.21.15.169:0/1729416864 learned_addr learned my addr 172.21.15.169:0/1729416864 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72de105700 1 -- 172.21.15.169:0/1729416864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d4005040 con 0x7f72d8060360 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72de105700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f69d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f72d4009f80 tx=0x7f72d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d4009320 con 0x7f72d8060360 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f72d4013680 con 0x7f72d8060360 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d80f50c0 con 0x7f72d8060360 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.617+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d401eda0 con 0x7f72d8060360 2024-01-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.618+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d80f5550 con 0x7f72d8060360 2024-01-11T05:06:54.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.618+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f72d401e420 con 0x7f72d8060360 2024-01-11T05:06:54.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.618+0000 7f72ceffd700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f72c4031090 0x7f72c4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:54.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.618+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d80591e0 con 0x7f72d8060360 2024-01-11T05:06:54.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.618+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f72d4042390 con 0x7f72d8060360 2024-01-11T05:06:54.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.619+0000 7f72dd904700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f72c4031090 0x7f72c4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:54.621 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.619+0000 7f72dd904700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f72c4031090 0x7f72c4033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f72c80096a0 tx=0x7f72c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:54.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.623+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f72d401e700 con 0x7f72d8060360 2024-01-11T05:06:54.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.832+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72d80fbec0 con 0x7f72d8060360 2024-01-11T05:06:54.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.832+0000 7f72ceffd700 1 -- 172.21.15.169:0/1729416864 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72d401cb10 con 0x7f72d8060360 2024-01-11T05:06:54.833 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:54.833 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f72c4031090 msgr2=0x7f72c4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f72c4031090 0x7f72c4033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f72c80096a0 tx=0x7f72c8006b40).stop 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 msgr2=0x7f72d80f69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f69d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f72d4009f80 tx=0x7f72d400b040).stop 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 shutdown_connections 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f72c4031090 0x7f72c4033550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 --2- 172.21.15.169:0/1729416864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72d8060360 0x7f72d80f69d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 >> 172.21.15.169:0/1729416864 conn(0x7f72d80f0a40 msgr2=0x7f72d80f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:54.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 shutdown_connections 2024-01-11T05:06:54.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:54.834+0000 7f72df107700 1 -- 172.21.15.169:0/1729416864 wait complete. 2024-01-11T05:06:54.836 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:54.301950+0000 mgr.smithi067.srbzpc (mgr.14156) 108 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:54.833595+0000 mon.smithi067 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.169:0/1729416864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:56.207 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:56.207 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:57.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1906577314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 msgr2=0x7f0d880f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 --2- 172.21.15.169:0/1906577314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d880f69b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f0d7c004760 tx=0x7f0d7c009ad0).stop 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1906577314 shutdown_connections 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 --2- 172.21.15.169:0/1906577314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d880f69b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1906577314 >> 172.21.15.169:0/1906577314 conn(0x7f0d880f1cd0 msgr2=0x7f0d880f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1906577314 shutdown_connections 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.321+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1906577314 wait complete. 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8ce77700 1 Processor -- start 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8ce77700 1 -- start start 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8ce77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d881089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:57.324 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8ce77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88108ed0 con 0x7f0d880f6590 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d881089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d881089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:36470/0 (socket says 172.21.15.169:36470) 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8659c700 1 -- 172.21.15.169:0/1791566450 learned_addr learned my addr 172.21.15.169:0/1791566450 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8659c700 1 -- 172.21.15.169:0/1791566450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c009d70 con 0x7f0d880f6590 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.322+0000 7f0d8659c700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d881089e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f0d7c004170 tx=0x7f0d7c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.323+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d7c00e040 con 0x7f0d880f6590 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.323+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d881090d0 con 0x7f0d880f6590 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.323+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0d7c008ef0 con 0x7f0d880f6590 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.323+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d7c014c90 con 0x7f0d880f6590 2024-01-11T05:06:57.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.323+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d88109560 con 0x7f0d880f6590 2024-01-11T05:06:57.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.323+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0d7c014420 con 0x7f0d880f6590 2024-01-11T05:06:57.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.324+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d88045bb0 con 0x7f0d880f6590 2024-01-11T05:06:57.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.324+0000 7f0d76ffd700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0d70031090 0x7f0d70033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:06:57.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.324+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0d7c042300 con 0x7f0d880f6590 2024-01-11T05:06:57.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.324+0000 7f0d85d9b700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0d70031090 0x7f0d70033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:06:57.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.324+0000 7f0d85d9b700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0d70031090 0x7f0d70033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f0d780096a0 tx=0x7f0d78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:06:57.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.329+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0d7c028c90 con 0x7f0d880f6590 2024-01-11T05:06:57.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.539+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d8810a410 con 0x7f0d880f6590 2024-01-11T05:06:57.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.540+0000 7f0d76ffd700 1 -- 172.21.15.169:0/1791566450 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d7c012b20 con 0x7f0d880f6590 2024-01-11T05:06:57.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:06:57.541 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0d70031090 msgr2=0x7f0d70033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0d70031090 0x7f0d70033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f0d780096a0 tx=0x7f0d78006b40).stop 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 msgr2=0x7f0d881089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d881089e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f0d7c004170 tx=0x7f0d7c009ad0).stop 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 shutdown_connections 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0d70031090 0x7f0d70033550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 --2- 172.21.15.169:0/1791566450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d880f6590 0x7f0d881089e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 >> 172.21.15.169:0/1791566450 conn(0x7f0d880f1cd0 msgr2=0x7f0d880f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 shutdown_connections 2024-01-11T05:06:57.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:06:57.542+0000 7f0d8ce77700 1 -- 172.21.15.169:0/1791566450 wait complete. 2024-01-11T05:06:57.544 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:06:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:56.302344+0000 mgr.smithi067.srbzpc (mgr.14156) 109 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:06:58.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:58 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:06:57.541083+0000 mon.smithi067 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.169:0/1791566450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:06:58.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:06:58.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:06:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:06:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:06:58.302669+0000 mgr.smithi067.srbzpc (mgr.14156) 110 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:00.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 -- 172.21.15.169:0/3756043093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 msgr2=0x7fb0940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:00.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 --2- 172.21.15.169:0/3756043093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0940f58a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fb088009fb0 tx=0x7fb088009b30).stop 2024-01-11T05:07:00.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 -- 172.21.15.169:0/3756043093 shutdown_connections 2024-01-11T05:07:00.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 --2- 172.21.15.169:0/3756043093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0940f58a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:00.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 -- 172.21.15.169:0/3756043093 >> 172.21.15.169:0/3756043093 conn(0x7fb0940f0be0 msgr2=0x7fb0940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:00.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 -- 172.21.15.169:0/3756043093 shutdown_connections 2024-01-11T05:07:00.006 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.004+0000 7fb098ef6700 1 -- 172.21.15.169:0/3756043093 wait complete. 2024-01-11T05:07:00.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.005+0000 7fb098ef6700 1 Processor -- start 2024-01-11T05:07:00.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.005+0000 7fb098ef6700 1 -- start start 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.005+0000 7fb098ef6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0941056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.005+0000 7fb098ef6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094105be0 con 0x7fb0940f5480 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb09259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0941056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb09259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0941056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59030/0 (socket says 172.21.15.169:59030) 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb09259c700 1 -- 172.21.15.169:0/3097257395 learned_addr learned my addr 172.21.15.169:0/3097257395 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb09259c700 1 -- 172.21.15.169:0/3097257395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088005040 con 0x7fb0940f5480 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb09259c700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0941056f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb0880089f0 tx=0x7fb08800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb088009320 con 0x7fb0940f5480 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094105de0 con 0x7fb0940f5480 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb088013ec0 con 0x7fb0940f5480 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb088014d30 con 0x7fb0940f5480 2024-01-11T05:07:00.008 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.006+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094106270 con 0x7fb0940f5480 2024-01-11T05:07:00.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.007+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb0880095f0 con 0x7fb0940f5480 2024-01-11T05:07:00.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.007+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094045bb0 con 0x7fb0940f5480 2024-01-11T05:07:00.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.007+0000 7fb082ffd700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb07c031090 0x7fb07c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:00.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.007+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb0880423e0 con 0x7fb0940f5480 2024-01-11T05:07:00.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.007+0000 7fb091d9b700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb07c031090 0x7fb07c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:00.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.008+0000 7fb091d9b700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb07c031090 0x7fb07c033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fb0840096a0 tx=0x7fb084006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:00.013 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.012+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb08801d7f0 con 0x7fb0940f5480 2024-01-11T05:07:00.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.223+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0940faf70 con 0x7fb0940f5480 2024-01-11T05:07:00.224 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.224+0000 7fb082ffd700 1 -- 172.21.15.169:0/3097257395 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb08801d7f0 con 0x7fb0940f5480 2024-01-11T05:07:00.225 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:00.225 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:00.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb07c031090 msgr2=0x7fb07c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:00.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb07c031090 0x7fb07c033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fb0840096a0 tx=0x7fb084006b40).stop 2024-01-11T05:07:00.226 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 msgr2=0x7fb0941056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0941056f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb0880089f0 tx=0x7fb08800b040).stop 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 shutdown_connections 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb07c031090 0x7fb07c033550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 --2- 172.21.15.169:0/3097257395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0940f5480 0x7fb0941056f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 >> 172.21.15.169:0/3097257395 conn(0x7fb0940f0be0 msgr2=0x7fb0940f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 shutdown_connections 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:00.225+0000 7fb098ef6700 1 -- 172.21.15.169:0/3097257395 wait complete. 2024-01-11T05:07:00.227 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:00.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:00 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:00.224817+0000 mon.smithi067 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.169:0/3097257395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:01.599 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:01.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:00.302901+0000 mgr.smithi067.srbzpc (mgr.14156) 111 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:02.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 -- 172.21.15.169:0/445775816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 msgr2=0x7fec200f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:02.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 --2- 172.21.15.169:0/445775816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec200f58d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fec08009fb0 tx=0x7fec08009b30).stop 2024-01-11T05:07:02.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 -- 172.21.15.169:0/445775816 shutdown_connections 2024-01-11T05:07:02.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 --2- 172.21.15.169:0/445775816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec200f58d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 -- 172.21.15.169:0/445775816 >> 172.21.15.169:0/445775816 conn(0x7fec200f0c10 msgr2=0x7fec200f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 -- 172.21.15.169:0/445775816 shutdown_connections 2024-01-11T05:07:02.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.733+0000 7fec2592a700 1 -- 172.21.15.169:0/445775816 wait complete. 2024-01-11T05:07:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec2592a700 1 Processor -- start 2024-01-11T05:07:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec2592a700 1 -- start start 2024-01-11T05:07:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec2592a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec201056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec2592a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20105b90 con 0x7fec200f54b0 2024-01-11T05:07:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec1effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec201056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:02.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec1effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec201056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59034/0 (socket says 172.21.15.169:59034) 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.734+0000 7fec1effd700 1 -- 172.21.15.169:0/3262845051 learned_addr learned my addr 172.21.15.169:0/3262845051 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec1effd700 1 -- 172.21.15.169:0/3262845051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08005040 con 0x7fec200f54b0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec1effd700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec201056a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fec080089f0 tx=0x7fec0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec08009320 con 0x7fec200f54b0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec20105d90 con 0x7fec200f54b0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fec08013ec0 con 0x7fec200f54b0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec08014d30 con 0x7fec200f54b0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.735+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec20106220 con 0x7fec200f54b0 2024-01-11T05:07:02.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.736+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fec080263a0 con 0x7fec200f54b0 2024-01-11T05:07:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.736+0000 7fec177fe700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fec0c031090 0x7fec0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.736+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fec08042440 con 0x7fec200f54b0 2024-01-11T05:07:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.736+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec20045bb0 con 0x7fec200f54b0 2024-01-11T05:07:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.736+0000 7fec1e7fc700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fec0c031090 0x7fec0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:02.738 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.736+0000 7fec1e7fc700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fec0c031090 0x7fec0c033550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fec100096a0 tx=0x7fec10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:02.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.741+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fec080266d0 con 0x7fec200f54b0 2024-01-11T05:07:02.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.950+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec20106ea0 con 0x7fec200f54b0 2024-01-11T05:07:02.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.951+0000 7fec177fe700 1 -- 172.21.15.169:0/3262845051 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec0802b020 con 0x7fec200f54b0 2024-01-11T05:07:02.952 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:02.952 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fec0c031090 msgr2=0x7fec0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fec0c031090 0x7fec0c033550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fec100096a0 tx=0x7fec10006b40).stop 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 msgr2=0x7fec201056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec201056a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fec080089f0 tx=0x7fec0800b040).stop 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 shutdown_connections 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fec0c031090 0x7fec0c033550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 --2- 172.21.15.169:0/3262845051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec200f54b0 0x7fec201056a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 >> 172.21.15.169:0/3262845051 conn(0x7fec200f0c10 msgr2=0x7fec200f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:02.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 shutdown_connections 2024-01-11T05:07:02.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:02.952+0000 7fec2592a700 1 -- 172.21.15.169:0/3262845051 wait complete. 2024-01-11T05:07:02.955 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:02.303213+0000 mgr.smithi067.srbzpc (mgr.14156) 112 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:02.952114+0000 mon.smithi067 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.169:0/3262845051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:04.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:04.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:05.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 -- 172.21.15.169:0/1645794151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 msgr2=0x7f94d40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:05.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 --2- 172.21.15.169:0/1645794151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d40f58a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f94c4009fb0 tx=0x7f94c4009b30).stop 2024-01-11T05:07:05.424 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 -- 172.21.15.169:0/1645794151 shutdown_connections 2024-01-11T05:07:05.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 --2- 172.21.15.169:0/1645794151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d40f58a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:05.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 -- 172.21.15.169:0/1645794151 >> 172.21.15.169:0/1645794151 conn(0x7f94d40f0be0 msgr2=0x7f94d40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:05.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 -- 172.21.15.169:0/1645794151 shutdown_connections 2024-01-11T05:07:05.425 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.423+0000 7f94d8b61700 1 -- 172.21.15.169:0/1645794151 wait complete. 2024-01-11T05:07:05.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d8b61700 1 Processor -- start 2024-01-11T05:07:05.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d8b61700 1 -- start start 2024-01-11T05:07:05.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d8b61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d4109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:05.426 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d8b61700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d4060300 con 0x7f94d40f5480 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d4109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d4109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59048/0 (socket says 172.21.15.169:59048) 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.424+0000 7f94d259c700 1 -- 172.21.15.169:0/2002761012 learned_addr learned my addr 172.21.15.169:0/2002761012 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94d259c700 1 -- 172.21.15.169:0/2002761012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4005040 con 0x7f94d40f5480 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94d259c700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d4109510 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f94c4000c00 tx=0x7f94c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94c40095f0 con 0x7f94d40f5480 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d4060500 con 0x7f94d40f5480 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f94c4007e50 con 0x7f94d40f5480 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94c4014e70 con 0x7f94d40f5480 2024-01-11T05:07:05.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.425+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d4109bc0 con 0x7f94d40f5480 2024-01-11T05:07:05.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.426+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f94c4025070 con 0x7f94d40f5480 2024-01-11T05:07:05.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.426+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d40591e0 con 0x7f94d40f5480 2024-01-11T05:07:05.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.426+0000 7f94caffd700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f94c00310e0 0x7f94c00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:05.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.426+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f94c4041dd0 con 0x7f94d40f5480 2024-01-11T05:07:05.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.426+0000 7f94d1d9b700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f94c00310e0 0x7f94c00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:05.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.427+0000 7f94d1d9b700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f94c00310e0 0x7f94c00335a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f94bc0096a0 tx=0x7f94bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:05.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.431+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f94c4046070 con 0x7f94d40f5480 2024-01-11T05:07:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:04.303488+0000 mgr.smithi067.srbzpc (mgr.14156) 113 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:05.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.642+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94d4045bb0 con 0x7f94d40f5480 2024-01-11T05:07:05.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.642+0000 7f94caffd700 1 -- 172.21.15.169:0/2002761012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94c400f4e0 con 0x7f94d40f5480 2024-01-11T05:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:05.643 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f94c00310e0 msgr2=0x7f94c00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f94c00310e0 0x7f94c00335a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f94bc0096a0 tx=0x7f94bc006b40).stop 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 msgr2=0x7f94d4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d4109510 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f94c4000c00 tx=0x7f94c400b040).stop 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 shutdown_connections 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f94c00310e0 0x7f94c00335a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 --2- 172.21.15.169:0/2002761012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94d40f5480 0x7f94d4109510 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:05.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 >> 172.21.15.169:0/2002761012 conn(0x7f94d40f0be0 msgr2=0x7f94d40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 shutdown_connections 2024-01-11T05:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:05.644+0000 7f94d8b61700 1 -- 172.21.15.169:0/2002761012 wait complete. 2024-01-11T05:07:05.646 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:06.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:06 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:05.643659+0000 mon.smithi067 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.169:0/2002761012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:07.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:07.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:07.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:06.303846+0000 mgr.smithi067.srbzpc (mgr.14156) 114 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 -- 172.21.15.169:0/1717450485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 msgr2=0x7f12840f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 --2- 172.21.15.169:0/1717450485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12840f6760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f1274009fb0 tx=0x7f1274009b30).stop 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 -- 172.21.15.169:0/1717450485 shutdown_connections 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 --2- 172.21.15.169:0/1717450485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12840f6760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 -- 172.21.15.169:0/1717450485 >> 172.21.15.169:0/1717450485 conn(0x7f12840f1aa0 msgr2=0x7f12840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 -- 172.21.15.169:0/1717450485 shutdown_connections 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 -- 172.21.15.169:0/1717450485 wait complete. 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.106+0000 7f12892dd700 1 Processor -- start 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f12892dd700 1 -- start start 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f12892dd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12841065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:08.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f12892dd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1284106ac0 con 0x7f12840f6340 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f1283fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12841065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f1283fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12841065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59070/0 (socket says 172.21.15.169:59070) 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f1283fff700 1 -- 172.21.15.169:0/1681917417 learned_addr learned my addr 172.21.15.169:0/1681917417 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f1283fff700 1 -- 172.21.15.169:0/1681917417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1274005040 con 0x7f12840f6340 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.107+0000 7f1283fff700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12841065d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f1274000c00 tx=0x7f127400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.108+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12740095f0 con 0x7f12840f6340 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.108+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1284106cc0 con 0x7f12840f6340 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.108+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1274007e50 con 0x7f12840f6340 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.108+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1274014e70 con 0x7f12840f6340 2024-01-11T05:07:08.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.108+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1284107150 con 0x7f12840f6340 2024-01-11T05:07:08.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.108+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1274025070 con 0x7f12840f6340 2024-01-11T05:07:08.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.109+0000 7f1280ff9700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1270031090 0x7f1270033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:08.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.109+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1274041d50 con 0x7f12840f6340 2024-01-11T05:07:08.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.109+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12840591e0 con 0x7f12840f6340 2024-01-11T05:07:08.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.109+0000 7f12837fe700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1270031090 0x7f1270033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:08.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.109+0000 7f12837fe700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1270031090 0x7f1270033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f126c0096a0 tx=0x7f126c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:08.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.113+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1274026350 con 0x7f12840f6340 2024-01-11T05:07:08.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.322+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1284045bb0 con 0x7f12840f6340 2024-01-11T05:07:08.323 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.323+0000 7f1280ff9700 1 -- 172.21.15.169:0/1681917417 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1274025300 con 0x7f12840f6340 2024-01-11T05:07:08.324 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:08.324 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:08.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1270031090 msgr2=0x7f1270033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:08.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1270031090 0x7f1270033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f126c0096a0 tx=0x7f126c006b40).stop 2024-01-11T05:07:08.325 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 msgr2=0x7f12841065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12841065d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f1274000c00 tx=0x7f127400b040).stop 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 shutdown_connections 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1270031090 0x7f1270033550 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 --2- 172.21.15.169:0/1681917417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12840f6340 0x7f12841065d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.324+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 >> 172.21.15.169:0/1681917417 conn(0x7f12840f1aa0 msgr2=0x7f12840f2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.325+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 shutdown_connections 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:08.325+0000 7f12892dd700 1 -- 172.21.15.169:0/1681917417 wait complete. 2024-01-11T05:07:08.326 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:08.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:08.323954+0000 mon.smithi067 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.169:0/1681917417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:09.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:09.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:08.304229+0000 mgr.smithi067.srbzpc (mgr.14156) 115 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:10.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- 172.21.15.169:0/1641902329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 msgr2=0x7f890c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:10.822 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 --2- 172.21.15.169:0/1641902329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c0f58a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f88fc009fb0 tx=0x7f88fc009b30).stop 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- 172.21.15.169:0/1641902329 shutdown_connections 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 --2- 172.21.15.169:0/1641902329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c0f58a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- 172.21.15.169:0/1641902329 >> 172.21.15.169:0/1641902329 conn(0x7f890c0f0be0 msgr2=0x7f890c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- 172.21.15.169:0/1641902329 shutdown_connections 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- 172.21.15.169:0/1641902329 wait complete. 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 Processor -- start 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- start start 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.820+0000 7f891376a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c060320 con 0x7f890c0f5480 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8911506700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8911506700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49888/0 (socket says 172.21.15.169:49888) 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8911506700 1 -- 172.21.15.169:0/2110419411 learned_addr learned my addr 172.21.15.169:0/2110419411 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:10.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8911506700 1 -- 172.21.15.169:0/2110419411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc005040 con 0x7f890c0f5480 2024-01-11T05:07:10.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8911506700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c1094f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f88fc0089f0 tx=0x7f88fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:10.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88fc009320 con 0x7f890c0f5480 2024-01-11T05:07:10.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c060520 con 0x7f890c0f5480 2024-01-11T05:07:10.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f88fc013ec0 con 0x7f890c0f5480 2024-01-11T05:07:10.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88fc014d30 con 0x7f890c0f5480 2024-01-11T05:07:10.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.821+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c109ba0 con 0x7f890c0f5480 2024-01-11T05:07:10.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.822+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f88fc0095f0 con 0x7f890c0f5480 2024-01-11T05:07:10.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.822+0000 7f8901ffb700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f88f8031090 0x7f88f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:10.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.822+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f88fc0424a0 con 0x7f890c0f5480 2024-01-11T05:07:10.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.822+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c045bb0 con 0x7f890c0f5480 2024-01-11T05:07:10.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.823+0000 7f8910d05700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f88f8031090 0x7f88f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:10.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.823+0000 7f8910d05700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f88f8031090 0x7f88f8033550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:10.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:10.827+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f88fc01d7f0 con 0x7f890c0f5480 2024-01-11T05:07:11.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.039+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f890c0fb060 con 0x7f890c0f5480 2024-01-11T05:07:11.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f8901ffb700 1 -- 172.21.15.169:0/2110419411 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f88fc01d7f0 con 0x7f890c0f5480 2024-01-11T05:07:11.041 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:11.041 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:11.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f88f8031090 msgr2=0x7f88f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f891376a700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f88f8031090 0x7f88f8033550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).stop 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 msgr2=0x7f890c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f891376a700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c1094f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f88fc0089f0 tx=0x7f88fc00b040).stop 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 shutdown_connections 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.041+0000 7f891376a700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f88f8031090 0x7f88f8033550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.042+0000 7f891376a700 1 --2- 172.21.15.169:0/2110419411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890c0f5480 0x7f890c1094f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.042+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 >> 172.21.15.169:0/2110419411 conn(0x7f890c0f0be0 msgr2=0x7f890c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.042+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 shutdown_connections 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:11.042+0000 7f891376a700 1 -- 172.21.15.169:0/2110419411 wait complete. 2024-01-11T05:07:11.043 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:10.304597+0000 mgr.smithi067.srbzpc (mgr.14156) 116 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:11.041261+0000 mon.smithi067 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.169:0/2110419411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:12.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:12.419 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 -- 172.21.15.169:0/807459090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 msgr2=0x7f05080f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 --2- 172.21.15.169:0/807459090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f05080f9200 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f04fc009fb0 tx=0x7f04fc009b30).stop 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 -- 172.21.15.169:0/807459090 shutdown_connections 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 --2- 172.21.15.169:0/807459090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f05080f9200 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 -- 172.21.15.169:0/807459090 >> 172.21.15.169:0/807459090 conn(0x7f05080f0be0 msgr2=0x7f05080f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 -- 172.21.15.169:0/807459090 shutdown_connections 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.552+0000 7f050eae3700 1 -- 172.21.15.169:0/807459090 wait complete. 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.553+0000 7f050eae3700 1 Processor -- start 2024-01-11T05:07:13.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.553+0000 7f050eae3700 1 -- start start 2024-01-11T05:07:13.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.553+0000 7f050eae3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f0508106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:13.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.553+0000 7f050eae3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0508106c20 con 0x7f05080602c0 2024-01-11T05:07:13.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050c87f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f0508106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:13.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050c87f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f0508106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49896/0 (socket says 172.21.15.169:49896) 2024-01-11T05:07:13.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050c87f700 1 -- 172.21.15.169:0/173370052 learned_addr learned my addr 172.21.15.169:0/173370052 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050c87f700 1 -- 172.21.15.169:0/173370052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc005040 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050c87f700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f0508106730 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f04fc013fd0 tx=0x7f04fc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04fc018070 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f04fc007b10 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0508106e20 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04fc01c3c0 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.554+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05081072b0 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.555+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f04fc007c80 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.555+0000 7f05057fa700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f04f80310e0 0x7f04f80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.555+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0508045bb0 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.555+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f04fc0429f0 con 0x7f05080602c0 2024-01-11T05:07:13.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.556+0000 7f0507fff700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f04f80310e0 0x7f04f80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:13.558 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.556+0000 7f0507fff700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f04f80310e0 0x7f04f80335a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f04f40096a0 tx=0x7f04f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:13.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.561+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f04fc01cb10 con 0x7f05080602c0 2024-01-11T05:07:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:12.304907+0000 mgr.smithi067.srbzpc (mgr.14156) 117 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:13.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.768+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05081080c0 con 0x7f05080602c0 2024-01-11T05:07:13.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.768+0000 7f05057fa700 1 -- 172.21.15.169:0/173370052 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04fc025b80 con 0x7f05080602c0 2024-01-11T05:07:13.770 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:13.770 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f04f80310e0 msgr2=0x7f04f80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f04f80310e0 0x7f04f80335a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f04f40096a0 tx=0x7f04f4006b40).stop 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 msgr2=0x7f0508106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f0508106730 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f04fc013fd0 tx=0x7f04fc009b30).stop 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 shutdown_connections 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f04f80310e0 0x7f04f80335a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 --2- 172.21.15.169:0/173370052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05080602c0 0x7f0508106730 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 >> 172.21.15.169:0/173370052 conn(0x7f05080f0be0 msgr2=0x7f05080f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:13.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 shutdown_connections 2024-01-11T05:07:13.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:13.770+0000 7f050eae3700 1 -- 172.21.15.169:0/173370052 wait complete. 2024-01-11T05:07:13.772 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:14.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:14 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:13.769725+0000 mon.smithi067 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.169:0/173370052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:15.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:15.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:15.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:14.305254+0000 mgr.smithi067.srbzpc (mgr.14156) 118 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 -- 172.21.15.169:0/3681783836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 msgr2=0x7fdfdc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 --2- 172.21.15.169:0/3681783836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc0f58a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fdfcc009fb0 tx=0x7fdfcc009b30).stop 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 -- 172.21.15.169:0/3681783836 shutdown_connections 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 --2- 172.21.15.169:0/3681783836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc0f58a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 -- 172.21.15.169:0/3681783836 >> 172.21.15.169:0/3681783836 conn(0x7fdfdc0f0be0 msgr2=0x7fdfdc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 -- 172.21.15.169:0/3681783836 shutdown_connections 2024-01-11T05:07:16.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.253+0000 7fdfe2988700 1 -- 172.21.15.169:0/3681783836 wait complete. 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfe2988700 1 Processor -- start 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfe2988700 1 -- start start 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfe2988700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfe2988700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc060320 con 0x7fdfdc0f5480 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49912/0 (socket says 172.21.15.169:49912) 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.254+0000 7fdfdbfff700 1 -- 172.21.15.169:0/3454018709 learned_addr learned my addr 172.21.15.169:0/3454018709 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfdbfff700 1 -- 172.21.15.169:0/3454018709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc005040 con 0x7fdfdc0f5480 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfdbfff700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc1094f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fdfcc000c00 tx=0x7fdfcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfcc0095f0 con 0x7fdfdc0f5480 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc060520 con 0x7fdfdc0f5480 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdfcc007e50 con 0x7fdfdc0f5480 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfcc014e70 con 0x7fdfdc0f5480 2024-01-11T05:07:16.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.255+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc109ba0 con 0x7fdfdc0f5480 2024-01-11T05:07:16.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.256+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdfcc025070 con 0x7fdfdc0f5480 2024-01-11T05:07:16.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.256+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfdc045bb0 con 0x7fdfdc0f5480 2024-01-11T05:07:16.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.256+0000 7fdfd8ff9700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdfc4031090 0x7fdfc4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:16.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.256+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdfcc041d60 con 0x7fdfdc0f5480 2024-01-11T05:07:16.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.256+0000 7fdfdb7fe700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdfc4031090 0x7fdfc4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:16.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.257+0000 7fdfdb7fe700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdfc4031090 0x7fdfc4033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdfd00096a0 tx=0x7fdfd0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:16.261 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.261+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdfcc014770 con 0x7fdfdc0f5480 2024-01-11T05:07:16.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.469+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfdc0faf70 con 0x7fdfdc0f5480 2024-01-11T05:07:16.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.469+0000 7fdfd8ff9700 1 -- 172.21.15.169:0/3454018709 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfcc014770 con 0x7fdfdc0f5480 2024-01-11T05:07:16.469 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:16.470 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdfc4031090 msgr2=0x7fdfc4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdfc4031090 0x7fdfc4033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdfd00096a0 tx=0x7fdfd0006b40).stop 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 msgr2=0x7fdfdc1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc1094f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fdfcc000c00 tx=0x7fdfcc00b040).stop 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 shutdown_connections 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdfc4031090 0x7fdfc4033550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 --2- 172.21.15.169:0/3454018709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfdc0f5480 0x7fdfdc1094f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 >> 172.21.15.169:0/3454018709 conn(0x7fdfdc0f0be0 msgr2=0x7fdfdc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 shutdown_connections 2024-01-11T05:07:16.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:16.471+0000 7fdfe2988700 1 -- 172.21.15.169:0/3454018709 wait complete. 2024-01-11T05:07:16.472 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:17.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:16.305575+0000 mgr.smithi067.srbzpc (mgr.14156) 119 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:17.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:16.469638+0000 mon.smithi067 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.169:0/3454018709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:17.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:17.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:18.969 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 -- 172.21.15.169:0/2197524943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 msgr2=0x7f4f240f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 --2- 172.21.15.169:0/2197524943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f240f69b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f4f14004760 tx=0x7f4f14009ad0).stop 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 -- 172.21.15.169:0/2197524943 shutdown_connections 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 --2- 172.21.15.169:0/2197524943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f240f69b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 -- 172.21.15.169:0/2197524943 >> 172.21.15.169:0/2197524943 conn(0x7f4f240f1cd0 msgr2=0x7f4f240f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 -- 172.21.15.169:0/2197524943 shutdown_connections 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.968+0000 7f4f2b673700 1 -- 172.21.15.169:0/2197524943 wait complete. 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2b673700 1 Processor -- start 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2b673700 1 -- start start 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2b673700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f241067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2b673700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f24106cb0 con 0x7f4f240f6590 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2940f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f241067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:18.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2940f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f241067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49922/0 (socket says 172.21.15.169:49922) 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2940f700 1 -- 172.21.15.169:0/3066913576 learned_addr learned my addr 172.21.15.169:0/3066913576 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2940f700 1 -- 172.21.15.169:0/3066913576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14009d70 con 0x7f4f240f6590 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.969+0000 7f4f2940f700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f241067c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4f14004170 tx=0x7f4f14009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.970+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f1400e040 con 0x7f4f240f6590 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.970+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f24106eb0 con 0x7f4f240f6590 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.970+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4f14008ef0 con 0x7f4f240f6590 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.970+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f14014c90 con 0x7f4f240f6590 2024-01-11T05:07:18.971 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.970+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f24107340 con 0x7f4f240f6590 2024-01-11T05:07:18.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.971+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4f14014420 con 0x7f4f240f6590 2024-01-11T05:07:18.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.971+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f24045bb0 con 0x7f4f240f6590 2024-01-11T05:07:18.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.971+0000 7f4f19ffb700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4f10031090 0x7f4f10033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:18.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.971+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4f14042260 con 0x7f4f240f6590 2024-01-11T05:07:18.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.971+0000 7f4f28c0e700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4f10031090 0x7f4f10033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:18.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.971+0000 7f4f28c0e700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4f10031090 0x7f4f10033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4f200096a0 tx=0x7f4f20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:18.975 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:18.975+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4f14026300 con 0x7f4f240f6590 2024-01-11T05:07:19.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.184+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f24108190 con 0x7f4f240f6590 2024-01-11T05:07:19.185 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.185+0000 7f4f19ffb700 1 -- 172.21.15.169:0/3066913576 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f14012b20 con 0x7f4f240f6590 2024-01-11T05:07:19.185 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:19.185 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4f10031090 msgr2=0x7f4f10033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4f10031090 0x7f4f10033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4f200096a0 tx=0x7f4f20006b40).stop 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 msgr2=0x7f4f241067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f241067c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4f14004170 tx=0x7f4f14009ad0).stop 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 shutdown_connections 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4f10031090 0x7f4f10033550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 --2- 172.21.15.169:0/3066913576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f240f6590 0x7f4f241067c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 >> 172.21.15.169:0/3066913576 conn(0x7f4f240f1cd0 msgr2=0x7f4f240f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 shutdown_connections 2024-01-11T05:07:19.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:19.187+0000 7f4f2b673700 1 -- 172.21.15.169:0/3066913576 wait complete. 2024-01-11T05:07:19.188 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:18.305931+0000 mgr.smithi067.srbzpc (mgr.14156) 120 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:19.185545+0000 mon.smithi067 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.169:0/3066913576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:20.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:20.579 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 -- 172.21.15.169:0/26124186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e540f6590 msgr2=0x7f9e540f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 --2- 172.21.15.169:0/26124186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e540f6590 0x7f9e540f69b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9e4c009a00 tx=0x7f9e4c004cb0).stop 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 -- 172.21.15.169:0/26124186 shutdown_connections 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 --2- 172.21.15.169:0/26124186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e540f6590 0x7f9e540f69b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 -- 172.21.15.169:0/26124186 >> 172.21.15.169:0/26124186 conn(0x7f9e540f1cd0 msgr2=0x7f9e540f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 -- 172.21.15.169:0/26124186 shutdown_connections 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.729+0000 7f9e5a010700 1 -- 172.21.15.169:0/26124186 wait complete. 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e5a010700 1 Processor -- start 2024-01-11T05:07:21.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e5a010700 1 -- start start 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e5a010700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 0x7f9e54106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e5a010700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54107180 con 0x7f9e54106870 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 0x7f9e54106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 0x7f9e54106c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42164/0 (socket says 172.21.15.169:42164) 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.730+0000 7f9e537fe700 1 -- 172.21.15.169:0/432903436 learned_addr learned my addr 172.21.15.169:0/432903436 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e537fe700 1 -- 172.21.15.169:0/432903436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c009640 con 0x7f9e54106870 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e537fe700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 0x7f9e54106c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9e4c009040 tx=0x7f9e4c004cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e4c019070 con 0x7f9e54106870 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9e4c0079a0 con 0x7f9e54106870 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e54107380 con 0x7f9e54106870 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e4c003a70 con 0x7f9e54106870 2024-01-11T05:07:21.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.731+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e54107ed0 con 0x7f9e54106870 2024-01-11T05:07:21.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.732+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9e4c025060 con 0x7f9e54106870 2024-01-11T05:07:21.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.732+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e540591e0 con 0x7f9e54106870 2024-01-11T05:07:21.733 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.732+0000 7f9e3bfff700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e3c0310e0 0x7f9e3c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:21.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.732+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9e4c042680 con 0x7f9e54106870 2024-01-11T05:07:21.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.732+0000 7f9e52ffd700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e3c0310e0 0x7f9e3c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:21.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.733+0000 7f9e52ffd700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e3c0310e0 0x7f9e3c0335a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f9e480096a0 tx=0x7f9e48006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:21.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.737+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9e4c027d30 con 0x7f9e54106870 2024-01-11T05:07:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:20.306148+0000 mgr.smithi067.srbzpc (mgr.14156) 121 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:21.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.944+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e54108670 con 0x7f9e54106870 2024-01-11T05:07:21.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.945+0000 7f9e3bfff700 1 -- 172.21.15.169:0/432903436 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e4c027b50 con 0x7f9e54106870 2024-01-11T05:07:21.945 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:21.945 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e3c0310e0 msgr2=0x7f9e3c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e3c0310e0 0x7f9e3c0335a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f9e480096a0 tx=0x7f9e48006b40).stop 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 msgr2=0x7f9e54106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 0x7f9e54106c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9e4c009040 tx=0x7f9e4c004cb0).stop 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 shutdown_connections 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9e3c0310e0 0x7f9e3c0335a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 --2- 172.21.15.169:0/432903436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e54106870 0x7f9e54106c90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:21.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 >> 172.21.15.169:0/432903436 conn(0x7f9e540f1cd0 msgr2=0x7f9e540f2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:21.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 shutdown_connections 2024-01-11T05:07:21.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:21.947+0000 7f9e5a010700 1 -- 172.21.15.169:0/432903436 wait complete. 2024-01-11T05:07:21.948 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:22.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:22 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:21.945454+0000 mon.smithi067 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.169:0/432903436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:23.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:23.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:22.306426+0000 mgr.smithi067.srbzpc (mgr.14156) 122 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:22.414861+0000 mon.smithi067 (mon.0) 193 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:07:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:22.415557+0000 mon.smithi067 (mon.0) 194 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:07:24.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 -- 172.21.15.169:0/1708308996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 msgr2=0x7fa6ec0f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:24.508 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 --2- 172.21.15.169:0/1708308996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec0f4c70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa6dc009fb0 tx=0x7fa6dc009b30).stop 2024-01-11T05:07:24.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 -- 172.21.15.169:0/1708308996 shutdown_connections 2024-01-11T05:07:24.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 --2- 172.21.15.169:0/1708308996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec0f4c70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:24.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 -- 172.21.15.169:0/1708308996 >> 172.21.15.169:0/1708308996 conn(0x7fa6ec0f0390 msgr2=0x7fa6ec0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:24.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 -- 172.21.15.169:0/1708308996 shutdown_connections 2024-01-11T05:07:24.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.507+0000 7fa6f34fd700 1 -- 172.21.15.169:0/1708308996 wait complete. 2024-01-11T05:07:24.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f34fd700 1 Processor -- start 2024-01-11T05:07:24.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f34fd700 1 -- start start 2024-01-11T05:07:24.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f34fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f34fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec106c90 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f1299700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f1299700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42174/0 (socket says 172.21.15.169:42174) 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.508+0000 7fa6f1299700 1 -- 172.21.15.169:0/3908512767 learned_addr learned my addr 172.21.15.169:0/3908512767 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6f1299700 1 -- 172.21.15.169:0/3908512767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc005040 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6f1299700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec1067a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa6dc000c00 tx=0x7fa6dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6dc0095f0 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa6dc007e50 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec106e90 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6dc014e70 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.509+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec107320 con 0x7fa6ec0f4850 2024-01-11T05:07:24.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.510+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa6dc025070 con 0x7fa6ec0f4850 2024-01-11T05:07:24.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.510+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec045bb0 con 0x7fa6ec0f4850 2024-01-11T05:07:24.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.510+0000 7fa6e1ffb700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa6d8031090 0x7fa6d8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:24.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.510+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa6dc041dd0 con 0x7fa6ec0f4850 2024-01-11T05:07:24.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.510+0000 7fa6f0a98700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa6d8031090 0x7fa6d8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:24.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.511+0000 7fa6f0a98700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa6d8031090 0x7fa6d8033550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa6e80096a0 tx=0x7fa6e8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:24.516 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.515+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa6dc00f4e0 con 0x7fa6ec0f4850 2024-01-11T05:07:24.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.722+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6ec108120 con 0x7fa6ec0f4850 2024-01-11T05:07:24.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.723+0000 7fa6e1ffb700 1 -- 172.21.15.169:0/3908512767 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6dc00f300 con 0x7fa6ec0f4850 2024-01-11T05:07:24.724 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:24.724 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.724+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa6d8031090 msgr2=0x7fa6d8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.724+0000 7fa6f34fd700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa6d8031090 0x7fa6d8033550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa6e80096a0 tx=0x7fa6e8006b40).stop 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.724+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 msgr2=0x7fa6ec1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.724+0000 7fa6f34fd700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec1067a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa6dc000c00 tx=0x7fa6dc00b040).stop 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.724+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 shutdown_connections 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.725+0000 7fa6f34fd700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa6d8031090 0x7fa6d8033550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.725+0000 7fa6f34fd700 1 --2- 172.21.15.169:0/3908512767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6ec0f4850 0x7fa6ec1067a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.725+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 >> 172.21.15.169:0/3908512767 conn(0x7fa6ec0f0390 msgr2=0x7fa6ec0f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.725+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 shutdown_connections 2024-01-11T05:07:24.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:24.725+0000 7fa6f34fd700 1 -- 172.21.15.169:0/3908512767 wait complete. 2024-01-11T05:07:24.727 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:24.306749+0000 mgr.smithi067.srbzpc (mgr.14156) 123 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:24.724235+0000 mon.smithi067 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.169:0/3908512767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:26.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:26.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 -- 172.21.15.169:0/538822888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 msgr2=0x7fde280f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 --2- 172.21.15.169:0/538822888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde280f58a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fde1c009fb0 tx=0x7fde1c009b30).stop 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 -- 172.21.15.169:0/538822888 shutdown_connections 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 --2- 172.21.15.169:0/538822888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde280f58a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 -- 172.21.15.169:0/538822888 >> 172.21.15.169:0/538822888 conn(0x7fde280f0be0 msgr2=0x7fde280f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 -- 172.21.15.169:0/538822888 shutdown_connections 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.225+0000 7fde2ed8b700 1 -- 172.21.15.169:0/538822888 wait complete. 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.226+0000 7fde2ed8b700 1 Processor -- start 2024-01-11T05:07:27.227 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.226+0000 7fde2ed8b700 1 -- start start 2024-01-11T05:07:27.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.226+0000 7fde2ed8b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde281094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:27.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.226+0000 7fde2ed8b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde28060320 con 0x7fde280f5480 2024-01-11T05:07:27.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2cb27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde281094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:27.229 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2cb27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde281094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42188/0 (socket says 172.21.15.169:42188) 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2cb27700 1 -- 172.21.15.169:0/820527013 learned_addr learned my addr 172.21.15.169:0/820527013 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2cb27700 1 -- 172.21.15.169:0/820527013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c005040 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2cb27700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde281094f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fde1c0089f0 tx=0x7fde1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde1c009320 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde28060520 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fde1c013ec0 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde1c014d30 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.227+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde28109ba0 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.228+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fde1c0095f0 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.228+0000 7fde257fa700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fde18031090 0x7fde18033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.228+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde28045bb0 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.228+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fde1c0423e0 con 0x7fde280f5480 2024-01-11T05:07:27.230 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.229+0000 7fde27fff700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fde18031090 0x7fde18033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:27.231 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.229+0000 7fde27fff700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fde18031090 0x7fde18033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fde140096a0 tx=0x7fde14006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:27.234 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.233+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fde1c00fe40 con 0x7fde280f5480 2024-01-11T05:07:27.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.441+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde280fb060 con 0x7fde280f5480 2024-01-11T05:07:27.442 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.442+0000 7fde257fa700 1 -- 172.21.15.169:0/820527013 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde1c00fe40 con 0x7fde280f5480 2024-01-11T05:07:27.442 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:27.443 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:27.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fde18031090 msgr2=0x7fde18033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:27.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fde18031090 0x7fde18033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fde140096a0 tx=0x7fde14006b40).stop 2024-01-11T05:07:27.444 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 msgr2=0x7fde281094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde281094f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fde1c0089f0 tx=0x7fde1c00b040).stop 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 shutdown_connections 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fde18031090 0x7fde18033550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 --2- 172.21.15.169:0/820527013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde280f5480 0x7fde281094f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 >> 172.21.15.169:0/820527013 conn(0x7fde280f0be0 msgr2=0x7fde280f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 shutdown_connections 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:27.443+0000 7fde2ed8b700 1 -- 172.21.15.169:0/820527013 wait complete. 2024-01-11T05:07:27.445 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:26.307123+0000 mgr.smithi067.srbzpc (mgr.14156) 124 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:28.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:28 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:27.442909+0000 mon.smithi067 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.169:0/820527013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:28.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:28.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:28.307411+0000 mgr.smithi067.srbzpc (mgr.14156) 125 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 -- 172.21.15.169:0/831717164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 msgr2=0x7f92ac0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 --2- 172.21.15.169:0/831717164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac0f58a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f92a0009fb0 tx=0x7f92a0009b30).stop 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 -- 172.21.15.169:0/831717164 shutdown_connections 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 --2- 172.21.15.169:0/831717164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac0f58a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 -- 172.21.15.169:0/831717164 >> 172.21.15.169:0/831717164 conn(0x7f92ac0f0be0 msgr2=0x7f92ac0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 -- 172.21.15.169:0/831717164 shutdown_connections 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.941+0000 7f92b31d3700 1 -- 172.21.15.169:0/831717164 wait complete. 2024-01-11T05:07:29.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b31d3700 1 Processor -- start 2024-01-11T05:07:29.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b31d3700 1 -- start start 2024-01-11T05:07:29.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b31d3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:29.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b31d3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac060300 con 0x7f92ac0f5480 2024-01-11T05:07:29.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b0f6f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:29.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b0f6f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42258/0 (socket says 172.21.15.169:42258) 2024-01-11T05:07:29.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.942+0000 7f92b0f6f700 1 -- 172.21.15.169:0/216648424 learned_addr learned my addr 172.21.15.169:0/216648424 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:29.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92b0f6f700 1 -- 172.21.15.169:0/216648424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0005040 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92b0f6f700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac109510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f92a00089f0 tx=0x7f92a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a0009320 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f92a0013ec0 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac060500 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a0014d30 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.943+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac109bc0 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.944+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f92a00095f0 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.944+0000 7f92a97fa700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f929c031090 0x7f929c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.944+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac0591e0 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.944+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f92a00423e0 con 0x7f92ac0f5480 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.944+0000 7f92abfff700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f929c031090 0x7f929c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:29.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.944+0000 7f92abfff700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f929c031090 0x7f929c033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:29.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:29.949+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f92a000fe40 con 0x7f92ac0f5480 2024-01-11T05:07:30.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.158+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92ac045bb0 con 0x7f92ac0f5480 2024-01-11T05:07:30.160 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.159+0000 7f92a97fa700 1 -- 172.21.15.169:0/216648424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92a000fe40 con 0x7f92ac0f5480 2024-01-11T05:07:30.160 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:30.160 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:30.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.160+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f929c031090 msgr2=0x7f929c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.160+0000 7f92b31d3700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f929c031090 0x7f929c033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).stop 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.160+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 msgr2=0x7f92ac109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.160+0000 7f92b31d3700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac109510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f92a00089f0 tx=0x7f92a000b040).stop 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.161+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 shutdown_connections 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.161+0000 7f92b31d3700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f929c031090 0x7f929c033550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.161+0000 7f92b31d3700 1 --2- 172.21.15.169:0/216648424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac0f5480 0x7f92ac109510 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.161+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 >> 172.21.15.169:0/216648424 conn(0x7f92ac0f0be0 msgr2=0x7f92ac0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.161+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 shutdown_connections 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:30.161+0000 7f92b31d3700 1 -- 172.21.15.169:0/216648424 wait complete. 2024-01-11T05:07:30.162 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:30.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:30 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:30.160023+0000 mon.smithi067 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.169:0/216648424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:31.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:31.540 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:30.307793+0000 mgr.smithi067.srbzpc (mgr.14156) 126 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 -- 172.21.15.169:0/1329460723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 msgr2=0x7fad180f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 --2- 172.21.15.169:0/1329460723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f9160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fad08009fb0 tx=0x7fad08009b30).stop 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 -- 172.21.15.169:0/1329460723 shutdown_connections 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 --2- 172.21.15.169:0/1329460723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f9160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 -- 172.21.15.169:0/1329460723 >> 172.21.15.169:0/1329460723 conn(0x7fad180f0be0 msgr2=0x7fad180f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 -- 172.21.15.169:0/1329460723 shutdown_connections 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 -- 172.21.15.169:0/1329460723 wait complete. 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.640+0000 7fad1e7b7700 1 Processor -- start 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad1e7b7700 1 -- start start 2024-01-11T05:07:32.643 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad1e7b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad1e7b7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad180f5060 con 0x7fad18060360 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad17fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad17fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42276/0 (socket says 172.21.15.169:42276) 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad17fff700 1 -- 172.21.15.169:0/3610437246 learned_addr learned my addr 172.21.15.169:0/3610437246 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad17fff700 1 -- 172.21.15.169:0/3610437246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad08005040 con 0x7fad18060360 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.641+0000 7fad17fff700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f6be0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fad08009f80 tx=0x7fad0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.642+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad08009320 con 0x7fad18060360 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.642+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad180f5200 con 0x7fad18060360 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.642+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad08013680 con 0x7fad18060360 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.642+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad0801eda0 con 0x7fad18060360 2024-01-11T05:07:32.644 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.642+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad180f5690 con 0x7fad18060360 2024-01-11T05:07:32.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.642+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fad0801e420 con 0x7fad18060360 2024-01-11T05:07:32.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.643+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad180591e0 con 0x7fad18060360 2024-01-11T05:07:32.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.643+0000 7fad14ff9700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fad04031090 0x7fad04033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:32.645 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.643+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fad08042420 con 0x7fad18060360 2024-01-11T05:07:32.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.643+0000 7fad177fe700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fad04031090 0x7fad04033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:32.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.643+0000 7fad177fe700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fad04031090 0x7fad04033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fad000096a0 tx=0x7fad00006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:32.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.648+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad08014cc0 con 0x7fad18060360 2024-01-11T05:07:32.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.857+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad180fc060 con 0x7fad18060360 2024-01-11T05:07:32.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.858+0000 7fad14ff9700 1 -- 172.21.15.169:0/3610437246 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad08004490 con 0x7fad18060360 2024-01-11T05:07:32.859 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:32.859 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fad04031090 msgr2=0x7fad04033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fad04031090 0x7fad04033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fad000096a0 tx=0x7fad00006b40).stop 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 msgr2=0x7fad180f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f6be0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fad08009f80 tx=0x7fad0800b040).stop 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 shutdown_connections 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fad04031090 0x7fad04033550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 --2- 172.21.15.169:0/3610437246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad18060360 0x7fad180f6be0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 >> 172.21.15.169:0/3610437246 conn(0x7fad180f0be0 msgr2=0x7fad180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 shutdown_connections 2024-01-11T05:07:32.861 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:32.860+0000 7fad1e7b7700 1 -- 172.21.15.169:0/3610437246 wait complete. 2024-01-11T05:07:32.862 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:32.308122+0000 mgr.smithi067.srbzpc (mgr.14156) 127 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:32.859421+0000 mon.smithi067 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.169:0/3610437246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:34.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:34.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:35.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 -- 172.21.15.169:0/3764118029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 msgr2=0x7f75040f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:35.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 --2- 172.21.15.169:0/3764118029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75040f5030 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f74f4009fb0 tx=0x7f74f4009b30).stop 2024-01-11T05:07:35.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 -- 172.21.15.169:0/3764118029 shutdown_connections 2024-01-11T05:07:35.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 --2- 172.21.15.169:0/3764118029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75040f5030 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:35.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 -- 172.21.15.169:0/3764118029 >> 172.21.15.169:0/3764118029 conn(0x7f75040f0390 msgr2=0x7f75040f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:35.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 -- 172.21.15.169:0/3764118029 shutdown_connections 2024-01-11T05:07:35.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.383+0000 7f750b858700 1 -- 172.21.15.169:0/3764118029 wait complete. 2024-01-11T05:07:35.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f750b858700 1 Processor -- start 2024-01-11T05:07:35.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f750b858700 1 -- start start 2024-01-11T05:07:35.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f750b858700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75041056d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:35.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f750b858700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504105bc0 con 0x7f75040f4c10 2024-01-11T05:07:35.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f75095f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75041056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:35.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f75095f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75041056d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42302/0 (socket says 172.21.15.169:42302) 2024-01-11T05:07:35.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.384+0000 7f75095f4700 1 -- 172.21.15.169:0/3755086889 learned_addr learned my addr 172.21.15.169:0/3755086889 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:35.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f75095f4700 1 -- 172.21.15.169:0/3755086889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4005040 con 0x7f75040f4c10 2024-01-11T05:07:35.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f75095f4700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75041056d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f74f40089f0 tx=0x7f74f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:35.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74f4009320 con 0x7f75040f4c10 2024-01-11T05:07:35.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7504105dc0 con 0x7f75040f4c10 2024-01-11T05:07:35.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f74f4013ec0 con 0x7f75040f4c10 2024-01-11T05:07:35.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74f4014d30 con 0x7f75040f4c10 2024-01-11T05:07:35.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.385+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7504106250 con 0x7f75040f4c10 2024-01-11T05:07:35.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.386+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f74f40095f0 con 0x7f75040f4c10 2024-01-11T05:07:35.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.386+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7504045bb0 con 0x7f75040f4c10 2024-01-11T05:07:35.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.386+0000 7f74f9ffb700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74f00310e0 0x7f74f00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:35.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.386+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f74f40424e0 con 0x7f75040f4c10 2024-01-11T05:07:35.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.386+0000 7f7508df3700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74f00310e0 0x7f74f00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:35.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.387+0000 7f7508df3700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74f00310e0 0x7f74f00335a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f75000096a0 tx=0x7f7500006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:35.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.391+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f74f401d7f0 con 0x7f75040f4c10 2024-01-11T05:07:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:34.308420+0000 mgr.smithi067.srbzpc (mgr.14156) 128 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:35.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.597+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75040fa700 con 0x7f75040f4c10 2024-01-11T05:07:35.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.598+0000 7f74f9ffb700 1 -- 172.21.15.169:0/3755086889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74f400fe40 con 0x7f75040f4c10 2024-01-11T05:07:35.600 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:35.600 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74f00310e0 msgr2=0x7f74f00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74f00310e0 0x7f74f00335a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f75000096a0 tx=0x7f7500006b40).stop 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 msgr2=0x7f75041056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75041056d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f74f40089f0 tx=0x7f74f400b040).stop 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 shutdown_connections 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74f00310e0 0x7f74f00335a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:35.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 --2- 172.21.15.169:0/3755086889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75040f4c10 0x7f75041056d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:35.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 >> 172.21.15.169:0/3755086889 conn(0x7f75040f0390 msgr2=0x7f75040f2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:35.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 shutdown_connections 2024-01-11T05:07:35.603 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:35.600+0000 7f750b858700 1 -- 172.21.15.169:0/3755086889 wait complete. 2024-01-11T05:07:35.603 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:36.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:35.599617+0000 mon.smithi067 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.169:0/3755086889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:36.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:36.985 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:37.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:36.308804+0000 mgr.smithi067.srbzpc (mgr.14156) 129 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:38.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 -- 172.21.15.169:0/4015559381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 msgr2=0x7fdb500f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 --2- 172.21.15.169:0/4015559381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb500f58a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fdb40009fb0 tx=0x7fdb40009b30).stop 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 -- 172.21.15.169:0/4015559381 shutdown_connections 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 --2- 172.21.15.169:0/4015559381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb500f58a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 -- 172.21.15.169:0/4015559381 >> 172.21.15.169:0/4015559381 conn(0x7fdb500f0be0 msgr2=0x7fdb500f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 -- 172.21.15.169:0/4015559381 shutdown_connections 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.066+0000 7fdb585a6700 1 -- 172.21.15.169:0/4015559381 wait complete. 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb585a6700 1 Processor -- start 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb585a6700 1 -- start start 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb585a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb50109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb585a6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50060300 con 0x7fdb500f5480 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb56342700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb50109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:38.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb56342700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb50109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42324/0 (socket says 172.21.15.169:42324) 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb56342700 1 -- 172.21.15.169:0/3831342831 learned_addr learned my addr 172.21.15.169:0/3831342831 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb56342700 1 -- 172.21.15.169:0/3831342831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb40005040 con 0x7fdb500f5480 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.067+0000 7fdb56342700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb50109510 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fdb400089f0 tx=0x7fdb4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.068+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb40009320 con 0x7fdb500f5480 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.068+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb50060500 con 0x7fdb500f5480 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.068+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdb40013ec0 con 0x7fdb500f5480 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.068+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb40014d30 con 0x7fdb500f5480 2024-01-11T05:07:38.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.068+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb50109bc0 con 0x7fdb500f5480 2024-01-11T05:07:38.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.068+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdb400095f0 con 0x7fdb500f5480 2024-01-11T05:07:38.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.069+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb500591e0 con 0x7fdb500f5480 2024-01-11T05:07:38.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.069+0000 7fdb46ffd700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdb3c0310e0 0x7fdb3c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:38.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.069+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdb40042450 con 0x7fdb500f5480 2024-01-11T05:07:38.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.069+0000 7fdb55b41700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdb3c0310e0 0x7fdb3c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:38.071 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.069+0000 7fdb55b41700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdb3c0310e0 0x7fdb3c0335a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdb4c0096a0 tx=0x7fdb4c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:38.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.074+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdb4001d980 con 0x7fdb500f5480 2024-01-11T05:07:38.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.280+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb50045bb0 con 0x7fdb500f5480 2024-01-11T05:07:38.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.281+0000 7fdb46ffd700 1 -- 172.21.15.169:0/3831342831 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb4001d380 con 0x7fdb500f5480 2024-01-11T05:07:38.281 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:38.282 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdb3c0310e0 msgr2=0x7fdb3c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdb3c0310e0 0x7fdb3c0335a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdb4c0096a0 tx=0x7fdb4c006b40).stop 2024-01-11T05:07:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 msgr2=0x7fdb50109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb50109510 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fdb400089f0 tx=0x7fdb4000b040).stop 2024-01-11T05:07:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 shutdown_connections 2024-01-11T05:07:38.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdb3c0310e0 0x7fdb3c0335a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:38.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 --2- 172.21.15.169:0/3831342831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb500f5480 0x7fdb50109510 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:38.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 >> 172.21.15.169:0/3831342831 conn(0x7fdb500f0be0 msgr2=0x7fdb500f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:38.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 shutdown_connections 2024-01-11T05:07:38.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:38.282+0000 7fdb585a6700 1 -- 172.21.15.169:0/3831342831 wait complete. 2024-01-11T05:07:38.284 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:38.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:38.281830+0000 mon.smithi067 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.169:0/3831342831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:39.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:39.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:38.309175+0000 mgr.smithi067.srbzpc (mgr.14156) 130 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:40.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 -- 172.21.15.169:0/2788357925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 msgr2=0x7f92a40f5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 --2- 172.21.15.169:0/2788357925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a40f5670 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9294009fb0 tx=0x7f9294009b30).stop 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 -- 172.21.15.169:0/2788357925 shutdown_connections 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 --2- 172.21.15.169:0/2788357925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a40f5670 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 -- 172.21.15.169:0/2788357925 >> 172.21.15.169:0/2788357925 conn(0x7f92a40f09b0 msgr2=0x7f92a40f2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 -- 172.21.15.169:0/2788357925 shutdown_connections 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.763+0000 7f92a3fff700 1 -- 172.21.15.169:0/2788357925 wait complete. 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a3fff700 1 Processor -- start 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a3fff700 1 -- start start 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a4105440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:40.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a3fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4105930 con 0x7f92a40f5250 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a4105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a4105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42734/0 (socket says 172.21.15.169:42734) 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a2ffd700 1 -- 172.21.15.169:0/2202162970 learned_addr learned my addr 172.21.15.169:0/2202162970 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a2ffd700 1 -- 172.21.15.169:0/2202162970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9294005040 con 0x7f92a40f5250 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.764+0000 7f92a2ffd700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a4105440 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f92940089f0 tx=0x7f929400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.765+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9294009320 con 0x7f92a40f5250 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.765+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a4105b30 con 0x7f92a40f5250 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.765+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9294013ec0 con 0x7f92a40f5250 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.765+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9294014d30 con 0x7f92a40f5250 2024-01-11T05:07:40.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.765+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4105fc0 con 0x7f92a40f5250 2024-01-11T05:07:40.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.765+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f92940095f0 con 0x7f92a40f5250 2024-01-11T05:07:40.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.766+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4045bb0 con 0x7f92a40f5250 2024-01-11T05:07:40.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.766+0000 7f928b7fe700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f928c031090 0x7f928c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:40.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.766+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f92940423e0 con 0x7f92a40f5250 2024-01-11T05:07:40.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.766+0000 7f92a27fc700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f928c031090 0x7f928c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:40.769 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.766+0000 7f92a27fc700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f928c031090 0x7f928c033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:40.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.770+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f929401d7f0 con 0x7f92a40f5250 2024-01-11T05:07:40.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.982+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92a4106e70 con 0x7f92a40f5250 2024-01-11T05:07:40.984 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.983+0000 7f928b7fe700 1 -- 172.21.15.169:0/2202162970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f929401d7f0 con 0x7f92a40f5250 2024-01-11T05:07:40.984 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:40.984 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:40.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.984+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f928c031090 msgr2=0x7f928c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.984+0000 7f92a3fff700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f928c031090 0x7f928c033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f92980096a0 tx=0x7f9298006b40).stop 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.984+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 msgr2=0x7f92a4105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.984+0000 7f92a3fff700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a4105440 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f92940089f0 tx=0x7f929400b040).stop 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.985+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 shutdown_connections 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.985+0000 7f92a3fff700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f928c031090 0x7f928c033550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.985+0000 7f92a3fff700 1 --2- 172.21.15.169:0/2202162970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92a40f5250 0x7f92a4105440 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.985+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 >> 172.21.15.169:0/2202162970 conn(0x7f92a40f09b0 msgr2=0x7f92a40f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.985+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 shutdown_connections 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:40.985+0000 7f92a3fff700 1 -- 172.21.15.169:0/2202162970 wait complete. 2024-01-11T05:07:40.986 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:40.309485+0000 mgr.smithi067.srbzpc (mgr.14156) 131 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:40.984157+0000 mon.smithi067 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.169:0/2202162970' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:42.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:42.369 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 -- 172.21.15.169:0/166488171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 msgr2=0x7f9f140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 --2- 172.21.15.169:0/166488171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f140f58a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9f08009fb0 tx=0x7f9f08009b30).stop 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 -- 172.21.15.169:0/166488171 shutdown_connections 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 --2- 172.21.15.169:0/166488171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f140f58a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 -- 172.21.15.169:0/166488171 >> 172.21.15.169:0/166488171 conn(0x7f9f140f0be0 msgr2=0x7f9f140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 -- 172.21.15.169:0/166488171 shutdown_connections 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 -- 172.21.15.169:0/166488171 wait complete. 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.471+0000 7f9f1ba06700 1 Processor -- start 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f1ba06700 1 -- start start 2024-01-11T05:07:43.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f1ba06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f141094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:43.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f1ba06700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14060320 con 0x7f9f140f5480 2024-01-11T05:07:43.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f197a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f141094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:43.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f197a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f141094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42758/0 (socket says 172.21.15.169:42758) 2024-01-11T05:07:43.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f197a2700 1 -- 172.21.15.169:0/1195042130 learned_addr learned my addr 172.21.15.169:0/1195042130 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:43.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f197a2700 1 -- 172.21.15.169:0/1195042130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08005040 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.472+0000 7f9f197a2700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f141094f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9f080089f0 tx=0x7f9f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.473+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f08009320 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.473+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f14060520 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.473+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9f08013ec0 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.473+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f08014d30 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.473+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f14109ba0 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.474+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9f08014420 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.474+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f14045bb0 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.474+0000 7f9f05ffb700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f000310e0 0x7f9f000335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.474+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9f080424a0 con 0x7f9f140f5480 2024-01-11T05:07:43.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.474+0000 7f9f18fa1700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f000310e0 0x7f9f000335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:43.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.475+0000 7f9f18fa1700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f000310e0 0x7f9f000335a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9f100096a0 tx=0x7f9f10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:43.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.479+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9f0801d7f0 con 0x7f9f140f5480 2024-01-11T05:07:43.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.687+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f140faf70 con 0x7f9f140f5480 2024-01-11T05:07:43.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.687+0000 7f9f05ffb700 1 -- 172.21.15.169:0/1195042130 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f0801d7f0 con 0x7f9f140f5480 2024-01-11T05:07:43.688 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:43.689 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f000310e0 msgr2=0x7f9f000335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f000310e0 0x7f9f000335a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9f100096a0 tx=0x7f9f10006b40).stop 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 msgr2=0x7f9f141094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f141094f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9f080089f0 tx=0x7f9f0800b040).stop 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 shutdown_connections 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f000310e0 0x7f9f000335a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 --2- 172.21.15.169:0/1195042130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f140f5480 0x7f9f141094f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 >> 172.21.15.169:0/1195042130 conn(0x7f9f140f0be0 msgr2=0x7f9f140f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 shutdown_connections 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:43.689+0000 7f9f1ba06700 1 -- 172.21.15.169:0/1195042130 wait complete. 2024-01-11T05:07:43.691 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:42.309838+0000 mgr.smithi067.srbzpc (mgr.14156) 132 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:44.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:44 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:43.688806+0000 mon.smithi067 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.169:0/1195042130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:45.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:45.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:45.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:44.310221+0000 mgr.smithi067.srbzpc (mgr.14156) 133 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 -- 172.21.15.169:0/4188788811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 msgr2=0x7f13cc0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 --2- 172.21.15.169:0/4188788811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f9160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f13bc009fb0 tx=0x7f13bc009b30).stop 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 -- 172.21.15.169:0/4188788811 shutdown_connections 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 --2- 172.21.15.169:0/4188788811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f9160 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 -- 172.21.15.169:0/4188788811 >> 172.21.15.169:0/4188788811 conn(0x7f13cc0f0be0 msgr2=0x7f13cc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 -- 172.21.15.169:0/4188788811 shutdown_connections 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.113+0000 7f13d4771700 1 -- 172.21.15.169:0/4188788811 wait complete. 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d4771700 1 Processor -- start 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d4771700 1 -- start start 2024-01-11T05:07:46.116 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d4771700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d4771700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc0f5060 con 0x7f13cc060360 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d250d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d250d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42782/0 (socket says 172.21.15.169:42782) 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d250d700 1 -- 172.21.15.169:0/113644812 learned_addr learned my addr 172.21.15.169:0/113644812 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d250d700 1 -- 172.21.15.169:0/113644812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc005040 con 0x7f13cc060360 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13d250d700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f6be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f13bc009f80 tx=0x7f13bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.114+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13bc009320 con 0x7f13cc060360 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.115+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc0f5200 con 0x7f13cc060360 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.115+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f13bc013680 con 0x7f13cc060360 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.115+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13bc01eda0 con 0x7f13cc060360 2024-01-11T05:07:46.117 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.115+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc0f5690 con 0x7f13cc060360 2024-01-11T05:07:46.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.115+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f13bc01e420 con 0x7f13cc060360 2024-01-11T05:07:46.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.116+0000 7f13c2ffd700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13b8031090 0x7f13b8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:46.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.116+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc0591e0 con 0x7f13cc060360 2024-01-11T05:07:46.118 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.116+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13bc042390 con 0x7f13cc060360 2024-01-11T05:07:46.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.116+0000 7f13d1d0c700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13b8031090 0x7f13b8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:46.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.116+0000 7f13d1d0c700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13b8031090 0x7f13b8033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f13c80096a0 tx=0x7f13c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:46.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.121+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f13bc01d380 con 0x7f13cc060360 2024-01-11T05:07:46.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.331+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13cc045bb0 con 0x7f13cc060360 2024-01-11T05:07:46.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.332+0000 7f13c2ffd700 1 -- 172.21.15.169:0/113644812 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13bc004490 con 0x7f13cc060360 2024-01-11T05:07:46.333 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:46.333 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:46.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.333+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13b8031090 msgr2=0x7f13b8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.333+0000 7f13d4771700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13b8031090 0x7f13b8033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f13c80096a0 tx=0x7f13c8006b40).stop 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.333+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 msgr2=0x7f13cc0f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.333+0000 7f13d4771700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f6be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f13bc009f80 tx=0x7f13bc00b040).stop 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.333+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 shutdown_connections 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.333+0000 7f13d4771700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13b8031090 0x7f13b8033550 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.334+0000 7f13d4771700 1 --2- 172.21.15.169:0/113644812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13cc060360 0x7f13cc0f6be0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.334+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 >> 172.21.15.169:0/113644812 conn(0x7f13cc0f0be0 msgr2=0x7f13cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.334+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 shutdown_connections 2024-01-11T05:07:46.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:46.334+0000 7f13d4771700 1 -- 172.21.15.169:0/113644812 wait complete. 2024-01-11T05:07:46.336 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:46.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:46.333105+0000 mon.smithi067 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.169:0/113644812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:47.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:47.706 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:46.310626+0000 mgr.smithi067.srbzpc (mgr.14156) 134 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:48.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 -- 172.21.15.169:0/742799557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 msgr2=0x7f9db80f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:48.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 --2- 172.21.15.169:0/742799557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db80f69a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f9da8009fb0 tx=0x7f9da8009b30).stop 2024-01-11T05:07:48.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 -- 172.21.15.169:0/742799557 shutdown_connections 2024-01-11T05:07:48.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 --2- 172.21.15.169:0/742799557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db80f69a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 -- 172.21.15.169:0/742799557 >> 172.21.15.169:0/742799557 conn(0x7f9db80f1ce0 msgr2=0x7f9db80f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 -- 172.21.15.169:0/742799557 shutdown_connections 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 -- 172.21.15.169:0/742799557 wait complete. 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 Processor -- start 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.825+0000 7f9dbe65e700 1 -- start start 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9dbe65e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:48.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9dbe65e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db8069f00 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9db7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9db7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42790/0 (socket says 172.21.15.169:42790) 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9db7fff700 1 -- 172.21.15.169:0/3946447080 learned_addr learned my addr 172.21.15.169:0/3946447080 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9db7fff700 1 -- 172.21.15.169:0/3946447080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8005040 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9db7fff700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db806ba80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9da80089f0 tx=0x7f9da800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da8009320 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.826+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db806a0a0 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.827+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9da8013ec0 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.827+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9da8014d30 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.827+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db806a530 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.827+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9da80263a0 con 0x7f9db80f6580 2024-01-11T05:07:48.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.827+0000 7f9db4ff9700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da0031090 0x7f9da0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:48.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.827+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9da8042430 con 0x7f9db80f6580 2024-01-11T05:07:48.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.828+0000 7f9db77fe700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da0031090 0x7f9da0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:48.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.828+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db80591e0 con 0x7f9db80f6580 2024-01-11T05:07:48.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.828+0000 7f9db77fe700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da0031090 0x7f9da0033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9dac0096a0 tx=0x7f9dac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:48.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:48.832+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9da8012920 con 0x7f9db80f6580 2024-01-11T05:07:49.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.040+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9db80fc070 con 0x7f9db80f6580 2024-01-11T05:07:49.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.040+0000 7f9db4ff9700 1 -- 172.21.15.169:0/3946447080 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9da802b020 con 0x7f9db80f6580 2024-01-11T05:07:49.041 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:49.042 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da0031090 msgr2=0x7f9da0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da0031090 0x7f9da0033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9dac0096a0 tx=0x7f9dac006b40).stop 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 msgr2=0x7f9db806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db806ba80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9da80089f0 tx=0x7f9da800b040).stop 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 shutdown_connections 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da0031090 0x7f9da0033550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:49.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 --2- 172.21.15.169:0/3946447080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db80f6580 0x7f9db806ba80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 >> 172.21.15.169:0/3946447080 conn(0x7f9db80f1ce0 msgr2=0x7f9db80f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 shutdown_connections 2024-01-11T05:07:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:49.042+0000 7f9dbe65e700 1 -- 172.21.15.169:0/3946447080 wait complete. 2024-01-11T05:07:49.044 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:48.310960+0000 mgr.smithi067.srbzpc (mgr.14156) 135 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:49.041698+0000 mon.smithi067 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.169:0/3946447080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:50.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:50.412 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/3354756840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 msgr2=0x7f5ac40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 --2- 172.21.15.169:0/3354756840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac40f69b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5ab8004760 tx=0x7f5ab8009ad0).stop 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/3354756840 shutdown_connections 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 --2- 172.21.15.169:0/3354756840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac40f69b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/3354756840 >> 172.21.15.169:0/3354756840 conn(0x7f5ac40f1cd0 msgr2=0x7f5ac40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/3354756840 shutdown_connections 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.520+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/3354756840 wait complete. 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac9d2c700 1 Processor -- start 2024-01-11T05:07:51.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac9d2c700 1 -- start start 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac9d2c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac9d2c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4106cb0 con 0x7f5ac40f6590 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43654/0 (socket says 172.21.15.169:43654) 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.521+0000 7f5ac37fe700 1 -- 172.21.15.169:0/849795839 learned_addr learned my addr 172.21.15.169:0/849795839 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5ac37fe700 1 -- 172.21.15.169:0/849795839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab8009d70 con 0x7f5ac40f6590 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5ac37fe700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac41067c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f5ab8004170 tx=0x7f5ab8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ab800e040 con 0x7f5ac40f6590 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac4106eb0 con 0x7f5ac40f6590 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5ab8008ef0 con 0x7f5ac40f6590 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ab8014c90 con 0x7f5ac40f6590 2024-01-11T05:07:51.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.522+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac4107340 con 0x7f5ac40f6590 2024-01-11T05:07:51.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.523+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5ab8014420 con 0x7f5ac40f6590 2024-01-11T05:07:51.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.523+0000 7f5aabfff700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5aac0310e0 0x7f5aac0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:51.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.523+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac4045bb0 con 0x7f5ac40f6590 2024-01-11T05:07:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.523+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5ab80421a0 con 0x7f5ac40f6590 2024-01-11T05:07:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.523+0000 7f5ac2ffd700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5aac0310e0 0x7f5aac0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:51.526 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.524+0000 7f5ac2ffd700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5aac0310e0 0x7f5aac0335a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5ab40096a0 tx=0x7f5ab4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:51.529 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.528+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5ab8026300 con 0x7f5ac40f6590 2024-01-11T05:07:51.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.738+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ac4108140 con 0x7f5ac40f6590 2024-01-11T05:07:51.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.739+0000 7f5aabfff700 1 -- 172.21.15.169:0/849795839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ab8012b20 con 0x7f5ac40f6590 2024-01-11T05:07:51.740 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:51.740 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5aac0310e0 msgr2=0x7f5aac0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5aac0310e0 0x7f5aac0335a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5ab40096a0 tx=0x7f5ab4006b40).stop 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 msgr2=0x7f5ac41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac41067c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f5ab8004170 tx=0x7f5ab8009ad0).stop 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 shutdown_connections 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5aac0310e0 0x7f5aac0335a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 --2- 172.21.15.169:0/849795839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40f6590 0x7f5ac41067c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 >> 172.21.15.169:0/849795839 conn(0x7f5ac40f1cd0 msgr2=0x7f5ac40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 shutdown_connections 2024-01-11T05:07:51.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:51.741+0000 7f5ac9d2c700 1 -- 172.21.15.169:0/849795839 wait complete. 2024-01-11T05:07:51.743 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:50.311291+0000 mgr.smithi067.srbzpc (mgr.14156) 136 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:52.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:51.739988+0000 mon.smithi067 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.169:0/849795839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:53.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:53.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:53.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:52.311681+0000 mgr.smithi067.srbzpc (mgr.14156) 137 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 -- 172.21.15.169:0/1179411669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 msgr2=0x7faaf00f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 --2- 172.21.15.169:0/1179411669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f9200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7faae4009fb0 tx=0x7faae4009b30).stop 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 -- 172.21.15.169:0/1179411669 shutdown_connections 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 --2- 172.21.15.169:0/1179411669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f9200 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 -- 172.21.15.169:0/1179411669 >> 172.21.15.169:0/1179411669 conn(0x7faaf00f0be0 msgr2=0x7faaf00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 -- 172.21.15.169:0/1179411669 shutdown_connections 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.211+0000 7faaf6fe4700 1 -- 172.21.15.169:0/1179411669 wait complete. 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf6fe4700 1 Processor -- start 2024-01-11T05:07:54.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf6fe4700 1 -- start start 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf6fe4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf6fe4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf00f5100 con 0x7faaf00602c0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf4d80700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf4d80700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43672/0 (socket says 172.21.15.169:43672) 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.212+0000 7faaf4d80700 1 -- 172.21.15.169:0/762096825 learned_addr learned my addr 172.21.15.169:0/762096825 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaf4d80700 1 -- 172.21.15.169:0/762096825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae4005040 con 0x7faaf00602c0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaf4d80700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f6c80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7faae4013fd0 tx=0x7faae4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae4018070 con 0x7faaf00602c0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf00f52a0 con 0x7faaf00602c0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faae4007b10 con 0x7faaf00602c0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faae401c3c0 con 0x7faaf00602c0 2024-01-11T05:07:54.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.213+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf00f5730 con 0x7faaf00602c0 2024-01-11T05:07:54.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.214+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7faae401d420 con 0x7faaf00602c0 2024-01-11T05:07:54.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.214+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf00591e0 con 0x7faaf00602c0 2024-01-11T05:07:54.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.214+0000 7faaed7fa700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7faae0031090 0x7faae0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:54.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.214+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faae4042980 con 0x7faaf00602c0 2024-01-11T05:07:54.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.214+0000 7faaeffff700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7faae0031090 0x7faae0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:54.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.215+0000 7faaeffff700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7faae0031090 0x7faae0033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7faadc0096a0 tx=0x7faadc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:54.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.219+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faae401b370 con 0x7faaf00602c0 2024-01-11T05:07:54.427 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.426+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faaf00fc070 con 0x7faaf00602c0 2024-01-11T05:07:54.428 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.427+0000 7faaed7fa700 1 -- 172.21.15.169:0/762096825 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faae401b370 con 0x7faaf00602c0 2024-01-11T05:07:54.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:54.429 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:54.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7faae0031090 msgr2=0x7faae0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:54.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7faae0031090 0x7faae0033550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7faadc0096a0 tx=0x7faadc006b40).stop 2024-01-11T05:07:54.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 msgr2=0x7faaf00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:54.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f6c80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7faae4013fd0 tx=0x7faae4009b30).stop 2024-01-11T05:07:54.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 shutdown_connections 2024-01-11T05:07:54.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7faae0031090 0x7faae0033550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:54.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 --2- 172.21.15.169:0/762096825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaf00602c0 0x7faaf00f6c80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:54.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 >> 172.21.15.169:0/762096825 conn(0x7faaf00f0be0 msgr2=0x7faaf00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:54.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 shutdown_connections 2024-01-11T05:07:54.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:54.429+0000 7faaf6fe4700 1 -- 172.21.15.169:0/762096825 wait complete. 2024-01-11T05:07:54.432 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:54.312101+0000 mgr.smithi067.srbzpc (mgr.14156) 138 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:54.428208+0000 mon.smithi067 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.169:0/762096825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:55.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:55.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 -- 172.21.15.169:0/588344594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 msgr2=0x7f9dbc0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 --2- 172.21.15.169:0/588344594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc0f58a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f9dac009fb0 tx=0x7f9dac009b30).stop 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 -- 172.21.15.169:0/588344594 shutdown_connections 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 --2- 172.21.15.169:0/588344594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc0f58a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 -- 172.21.15.169:0/588344594 >> 172.21.15.169:0/588344594 conn(0x7f9dbc0f0be0 msgr2=0x7f9dbc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 -- 172.21.15.169:0/588344594 shutdown_connections 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.894+0000 7f9dc2a27700 1 -- 172.21.15.169:0/588344594 wait complete. 2024-01-11T05:07:56.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.895+0000 7f9dc2a27700 1 Processor -- start 2024-01-11T05:07:56.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.895+0000 7f9dc2a27700 1 -- start start 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.895+0000 7f9dc2a27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.895+0000 7f9dc2a27700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc105be0 con 0x7f9dbc0f5480 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9dbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9dbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43696/0 (socket says 172.21.15.169:43696) 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9dbbfff700 1 -- 172.21.15.169:0/1574211614 learned_addr learned my addr 172.21.15.169:0/1574211614 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9dbbfff700 1 -- 172.21.15.169:0/1574211614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac005040 con 0x7f9dbc0f5480 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9dbbfff700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc1056f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9dac0089f0 tx=0x7f9dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dac009320 con 0x7f9dbc0f5480 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc105de0 con 0x7f9dbc0f5480 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9dac013ec0 con 0x7f9dbc0f5480 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.896+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dac014d30 con 0x7f9dbc0f5480 2024-01-11T05:07:56.898 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.897+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc106270 con 0x7f9dbc0f5480 2024-01-11T05:07:56.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.897+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9dac0263a0 con 0x7f9dbc0f5480 2024-01-11T05:07:56.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.898+0000 7f9db8ff9700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da4031090 0x7f9da4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:56.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.898+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9dac0423e0 con 0x7f9dbc0f5480 2024-01-11T05:07:56.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.898+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc0591e0 con 0x7f9dbc0f5480 2024-01-11T05:07:56.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.898+0000 7f9dbb7fe700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da4031090 0x7f9da4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:56.900 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.898+0000 7f9dbb7fe700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da4031090 0x7f9da4033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9db00096a0 tx=0x7f9db0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:56.903 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:56.903+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9dac0266d0 con 0x7f9dbc0f5480 2024-01-11T05:07:57.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.109+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dbc0faf70 con 0x7f9dbc0f5480 2024-01-11T05:07:57.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.110+0000 7f9db8ff9700 1 -- 172.21.15.169:0/1574211614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9dac02b020 con 0x7f9dbc0f5480 2024-01-11T05:07:57.111 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:57.111 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da4031090 msgr2=0x7f9da4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da4031090 0x7f9da4033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9db00096a0 tx=0x7f9db0006b40).stop 2024-01-11T05:07:57.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 msgr2=0x7f9dbc1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc1056f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9dac0089f0 tx=0x7f9dac00b040).stop 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 shutdown_connections 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9da4031090 0x7f9da4033550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 --2- 172.21.15.169:0/1574211614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc0f5480 0x7f9dbc1056f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 >> 172.21.15.169:0/1574211614 conn(0x7f9dbc0f0be0 msgr2=0x7f9dbc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 shutdown_connections 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:57.111+0000 7f9dc2a27700 1 -- 172.21.15.169:0/1574211614 wait complete. 2024-01-11T05:07:57.113 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:07:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:56.312448+0000 mgr.smithi067.srbzpc (mgr.14156) 139 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:57.110998+0000 mon.smithi067 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.169:0/1574211614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:07:58.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:07:58.488 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:07:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 -- 172.21.15.169:0/1894417104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 msgr2=0x7faba00f5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 --2- 172.21.15.169:0/1894417104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba00f5ce0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fab88009fb0 tx=0x7fab88009b30).stop 2024-01-11T05:07:59.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 -- 172.21.15.169:0/1894417104 shutdown_connections 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 --2- 172.21.15.169:0/1894417104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba00f5ce0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 -- 172.21.15.169:0/1894417104 >> 172.21.15.169:0/1894417104 conn(0x7faba00f01d0 msgr2=0x7faba00f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 -- 172.21.15.169:0/1894417104 shutdown_connections 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 -- 172.21.15.169:0/1894417104 wait complete. 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.589+0000 7faba524d700 1 Processor -- start 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7faba524d700 1 -- start start 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7faba524d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba0109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:59.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7faba524d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba00f5280 con 0x7faba00f58c0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7fab9ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba0109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7fab9ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba0109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56100/0 (socket says 172.21.15.169:56100) 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7fab9ffff700 1 -- 172.21.15.169:0/1419974728 learned_addr learned my addr 172.21.15.169:0/1419974728 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7fab9ffff700 1 -- 172.21.15.169:0/1419974728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88005040 con 0x7faba00f58c0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7fab9ffff700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba0109310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fab880089f0 tx=0x7fab8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.590+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab88009320 con 0x7faba00f58c0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.591+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba00f5480 con 0x7faba00f58c0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.591+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fab88013ec0 con 0x7faba00f58c0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.591+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba01099c0 con 0x7faba00f58c0 2024-01-11T05:07:59.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.591+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab88014d30 con 0x7faba00f58c0 2024-01-11T05:07:59.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.591+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fab88014420 con 0x7faba00f58c0 2024-01-11T05:07:59.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.591+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84004fa0 con 0x7faba00f58c0 2024-01-11T05:07:59.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.592+0000 7fab9cff9700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fab8c031090 0x7fab8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:07:59.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.592+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fab88042350 con 0x7faba00f58c0 2024-01-11T05:07:59.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.592+0000 7fab9f7fe700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fab8c031090 0x7fab8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:07:59.594 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.592+0000 7fab9f7fe700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fab8c031090 0x7fab8c033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fab900096a0 tx=0x7fab90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:07:59.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.596+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fab880268c0 con 0x7faba00f58c0 2024-01-11T05:07:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:07:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:07:58.312804+0000 mgr.smithi067.srbzpc (mgr.14156) 140 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:07:59.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.804+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab84005670 con 0x7faba00f58c0 2024-01-11T05:07:59.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.805+0000 7fab9cff9700 1 -- 172.21.15.169:0/1419974728 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab8801d7f0 con 0x7faba00f58c0 2024-01-11T05:07:59.806 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:07:59.806 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:07:59.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fab8c031090 msgr2=0x7fab8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:59.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fab8c031090 0x7fab8c033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fab900096a0 tx=0x7fab90006b40).stop 2024-01-11T05:07:59.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 msgr2=0x7faba0109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:07:59.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba0109310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fab880089f0 tx=0x7fab8800b040).stop 2024-01-11T05:07:59.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 shutdown_connections 2024-01-11T05:07:59.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fab8c031090 0x7fab8c033550 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:59.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 --2- 172.21.15.169:0/1419974728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba00f58c0 0x7faba0109310 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:07:59.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 >> 172.21.15.169:0/1419974728 conn(0x7faba00f01d0 msgr2=0x7faba00f0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:07:59.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 shutdown_connections 2024-01-11T05:07:59.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:07:59.806+0000 7faba524d700 1 -- 172.21.15.169:0/1419974728 wait complete. 2024-01-11T05:07:59.808 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:00.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:00 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:07:59.805974+0000 mon.smithi067 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.169:0/1419974728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:01.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:01.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:00.313213+0000 mgr.smithi067.srbzpc (mgr.14156) 141 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:02.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.298+0000 7f6945f27700 1 -- 172.21.15.169:0/3991622493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 msgr2=0x7f69400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:02.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.298+0000 7f6945f27700 1 --2- 172.21.15.169:0/3991622493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69400f58a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f6928009fb0 tx=0x7f6928009b30).stop 2024-01-11T05:08:02.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 -- 172.21.15.169:0/3991622493 shutdown_connections 2024-01-11T05:08:02.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 --2- 172.21.15.169:0/3991622493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69400f58a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:02.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 -- 172.21.15.169:0/3991622493 >> 172.21.15.169:0/3991622493 conn(0x7f69400f0be0 msgr2=0x7f69400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:02.300 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 -- 172.21.15.169:0/3991622493 shutdown_connections 2024-01-11T05:08:02.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 -- 172.21.15.169:0/3991622493 wait complete. 2024-01-11T05:08:02.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 Processor -- start 2024-01-11T05:08:02.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 -- start start 2024-01-11T05:08:02.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69401067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:02.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.299+0000 7f6945f27700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6940106ce0 con 0x7f69400f5480 2024-01-11T05:08:02.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f693f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69401067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:02.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f693f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69401067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56106/0 (socket says 172.21.15.169:56106) 2024-01-11T05:08:02.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f693f7fe700 1 -- 172.21.15.169:0/1797321190 learned_addr learned my addr 172.21.15.169:0/1797321190 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f693f7fe700 1 -- 172.21.15.169:0/1797321190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928005040 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f693f7fe700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69401067f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f69280089f0 tx=0x7f692800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6928009320 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6940106ee0 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6928013ec0 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.300+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6928014d30 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.301+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6940107370 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.301+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f69280263a0 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.301+0000 7f6937fff700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f692c031090 0x7f692c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.301+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f69280424e0 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.302+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69400591e0 con 0x7f69400f5480 2024-01-11T05:08:02.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.302+0000 7f693effd700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f692c031090 0x7f692c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:02.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.302+0000 7f693effd700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f692c031090 0x7f692c033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f69300096a0 tx=0x7f6930006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:02.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.307+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f69280266d0 con 0x7f69400f5480 2024-01-11T05:08:02.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.513+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69400faf70 con 0x7f69400f5480 2024-01-11T05:08:02.515 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.514+0000 7f6937fff700 1 -- 172.21.15.169:0/1797321190 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f692802b020 con 0x7f69400f5480 2024-01-11T05:08:02.515 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:02.515 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.515+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f692c031090 msgr2=0x7f692c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f692c031090 0x7f692c033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f69300096a0 tx=0x7f6930006b40).stop 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 msgr2=0x7f69401067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69401067f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f69280089f0 tx=0x7f692800b040).stop 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 shutdown_connections 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f692c031090 0x7f692c033550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 --2- 172.21.15.169:0/1797321190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69400f5480 0x7f69401067f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 >> 172.21.15.169:0/1797321190 conn(0x7f69400f0be0 msgr2=0x7f69400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:02.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 shutdown_connections 2024-01-11T05:08:02.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:02.516+0000 7f6945f27700 1 -- 172.21.15.169:0/1797321190 wait complete. 2024-01-11T05:08:02.518 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:02.313692+0000 mgr.smithi067.srbzpc (mgr.14156) 142 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:02.515108+0000 mon.smithi067 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.169:0/1797321190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:03.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:03.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 -- 172.21.15.169:0/1285183562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 msgr2=0x7f5f900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 --2- 172.21.15.169:0/1285183562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f900f58a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f5f80009fb0 tx=0x7f5f80009b30).stop 2024-01-11T05:08:05.004 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 -- 172.21.15.169:0/1285183562 shutdown_connections 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 --2- 172.21.15.169:0/1285183562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f900f58a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 -- 172.21.15.169:0/1285183562 >> 172.21.15.169:0/1285183562 conn(0x7f5f900f0be0 msgr2=0x7f5f900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 -- 172.21.15.169:0/1285183562 shutdown_connections 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 -- 172.21.15.169:0/1285183562 wait complete. 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.003+0000 7f5f95e2c700 1 Processor -- start 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f95e2c700 1 -- start start 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f95e2c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f901056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f95e2c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f90105be0 con 0x7f5f900f5480 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f8f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f901056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f8f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f901056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56120/0 (socket says 172.21.15.169:56120) 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f8f7fe700 1 -- 172.21.15.169:0/4007840051 learned_addr learned my addr 172.21.15.169:0/4007840051 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f8f7fe700 1 -- 172.21.15.169:0/4007840051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80005040 con 0x7f5f900f5480 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.004+0000 7f5f8f7fe700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f901056f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f5f800097f0 tx=0x7f5f8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:05.005 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.005+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f800095f0 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.005+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f90105de0 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.005+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5f80008c00 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.005+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f80014420 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.005+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f90106270 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.005+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5f8001c440 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.006+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f90045bb0 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.006+0000 7f5f77fff700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5f78031090 0x7f5f78033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.006+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5f800449c0 con 0x7f5f900f5480 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.006+0000 7f5f8effd700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5f78031090 0x7f5f78033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:05.007 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.007+0000 7f5f8effd700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5f78031090 0x7f5f78033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5f840096a0 tx=0x7f5f84006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:05.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.011+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5f80012330 con 0x7f5f900f5480 2024-01-11T05:08:05.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.218+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f900faf70 con 0x7f5f900f5480 2024-01-11T05:08:05.219 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.219+0000 7f5f77fff700 1 -- 172.21.15.169:0/4007840051 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f80012330 con 0x7f5f900f5480 2024-01-11T05:08:05.220 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:05.220 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:05.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5f78031090 msgr2=0x7f5f78033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:05.221 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5f78031090 0x7f5f78033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5f840096a0 tx=0x7f5f84006b40).stop 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 msgr2=0x7f5f901056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f901056f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f5f800097f0 tx=0x7f5f8000b040).stop 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 shutdown_connections 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5f78031090 0x7f5f78033550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 --2- 172.21.15.169:0/4007840051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f900f5480 0x7f5f901056f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 >> 172.21.15.169:0/4007840051 conn(0x7f5f900f0be0 msgr2=0x7f5f900f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 shutdown_connections 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:05.221+0000 7f5f95e2c700 1 -- 172.21.15.169:0/4007840051 wait complete. 2024-01-11T05:08:05.222 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:04.314049+0000 mgr.smithi067.srbzpc (mgr.14156) 143 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:06.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:06 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:05.219644+0000 mon.smithi067 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.169:0/4007840051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:06.592 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:06.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:07.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:06.314328+0000 mgr.smithi067.srbzpc (mgr.14156) 144 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2471616865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 msgr2=0x7f43a00f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 --2- 172.21.15.169:0/2471616865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a00f9200 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f439c009fb0 tx=0x7f439c009b30).stop 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2471616865 shutdown_connections 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 --2- 172.21.15.169:0/2471616865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a00f9200 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2471616865 >> 172.21.15.169:0/2471616865 conn(0x7f43a00f0be0 msgr2=0x7f43a00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2471616865 shutdown_connections 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.680+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2471616865 wait complete. 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a7c1d700 1 Processor -- start 2024-01-11T05:08:07.681 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a7c1d700 1 -- start start 2024-01-11T05:08:07.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a7c1d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:07.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a7c1d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a0106c20 con 0x7f43a00602c0 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a59b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a59b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56134/0 (socket says 172.21.15.169:56134) 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.681+0000 7f43a59b9700 1 -- 172.21.15.169:0/2396307942 learned_addr learned my addr 172.21.15.169:0/2396307942 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43a59b9700 1 -- 172.21.15.169:0/2396307942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c005040 con 0x7f43a00602c0 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43a59b9700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a0106730 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f439c013fd0 tx=0x7f439c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f439c018070 con 0x7f43a00602c0 2024-01-11T05:08:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a0106e20 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f439c007b10 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a01072b0 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.682+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f439c01c3c0 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.683+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f439c007c80 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.683+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a0045bb0 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.683+0000 7f43967fc700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f438c031090 0x7f438c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.683+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f439c042a10 con 0x7f43a00602c0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.683+0000 7f43a51b8700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f438c031090 0x7f438c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.684+0000 7f43a51b8700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f438c031090 0x7f438c033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f43900096a0 tx=0x7f4390006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:07.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.688+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f439c01b370 con 0x7f43a00602c0 2024-01-11T05:08:07.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.893+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43a0108100 con 0x7f43a00602c0 2024-01-11T05:08:07.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.894+0000 7f43967fc700 1 -- 172.21.15.169:0/2396307942 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f439c025b40 con 0x7f43a00602c0 2024-01-11T05:08:07.894 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:07.894 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:07.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f438c031090 msgr2=0x7f438c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:07.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f438c031090 0x7f438c033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f43900096a0 tx=0x7f4390006b40).stop 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 msgr2=0x7f43a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a0106730 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f439c013fd0 tx=0x7f439c009b30).stop 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 shutdown_connections 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f438c031090 0x7f438c033550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 --2- 172.21.15.169:0/2396307942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43a00602c0 0x7f43a0106730 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 >> 172.21.15.169:0/2396307942 conn(0x7f43a00f0be0 msgr2=0x7f43a00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 shutdown_connections 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:07.895+0000 7f43a7c1d700 1 -- 172.21.15.169:0/2396307942 wait complete. 2024-01-11T05:08:07.896 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:08.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:07.894077+0000 mon.smithi067 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.169:0/2396307942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:09.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:09.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:09.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:08.314599+0000 mgr.smithi067.srbzpc (mgr.14156) 145 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:10.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 -- 172.21.15.169:0/3865684781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 msgr2=0x7fc0600f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 --2- 172.21.15.169:0/3865684781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc0600f58d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc054009fb0 tx=0x7fc054009b30).stop 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 -- 172.21.15.169:0/3865684781 shutdown_connections 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 --2- 172.21.15.169:0/3865684781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc0600f58d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 -- 172.21.15.169:0/3865684781 >> 172.21.15.169:0/3865684781 conn(0x7fc0600f0c10 msgr2=0x7fc0600f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 -- 172.21.15.169:0/3865684781 shutdown_connections 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 -- 172.21.15.169:0/3865684781 wait complete. 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.389+0000 7fc0668ef700 1 Processor -- start 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc0668ef700 1 -- start start 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc0668ef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc060106840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:10.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc0668ef700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060106d30 con 0x7fc0600f54b0 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc060106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc060106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45496/0 (socket says 172.21.15.169:45496) 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc05ffff700 1 -- 172.21.15.169:0/922387553 learned_addr learned my addr 172.21.15.169:0/922387553 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc05ffff700 1 -- 172.21.15.169:0/922387553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054005040 con 0x7fc0600f54b0 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.390+0000 7fc05ffff700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc060106840 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fc054009f80 tx=0x7fc05400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.391+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc054009320 con 0x7fc0600f54b0 2024-01-11T05:08:10.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.391+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc060106f30 con 0x7fc0600f54b0 2024-01-11T05:08:10.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.391+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc054013380 con 0x7fc0600f54b0 2024-01-11T05:08:10.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.391+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc05401ebd0 con 0x7fc0600f54b0 2024-01-11T05:08:10.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.391+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0601073c0 con 0x7fc0600f54b0 2024-01-11T05:08:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.391+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc05401e420 con 0x7fc0600f54b0 2024-01-11T05:08:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.392+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060045bb0 con 0x7fc0600f54b0 2024-01-11T05:08:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.392+0000 7fc05cff9700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc048031090 0x7fc048033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.392+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc054042260 con 0x7fc0600f54b0 2024-01-11T05:08:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.392+0000 7fc05f7fe700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc048031090 0x7fc048033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:10.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.392+0000 7fc05f7fe700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc048031090 0x7fc048033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:10.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.397+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc05401d380 con 0x7fc0600f54b0 2024-01-11T05:08:10.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.603+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc060108270 con 0x7fc0600f54b0 2024-01-11T05:08:10.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.604+0000 7fc05cff9700 1 -- 172.21.15.169:0/922387553 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc05401cb10 con 0x7fc0600f54b0 2024-01-11T05:08:10.605 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:10.605 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc048031090 msgr2=0x7fc048033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc048031090 0x7fc048033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).stop 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 msgr2=0x7fc060106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc060106840 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fc054009f80 tx=0x7fc05400b040).stop 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 shutdown_connections 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc048031090 0x7fc048033550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 --2- 172.21.15.169:0/922387553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0600f54b0 0x7fc060106840 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.605+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 >> 172.21.15.169:0/922387553 conn(0x7fc0600f0c10 msgr2=0x7fc06005ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.606+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 shutdown_connections 2024-01-11T05:08:10.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:10.606+0000 7fc0668ef700 1 -- 172.21.15.169:0/922387553 wait complete. 2024-01-11T05:08:10.608 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:10.314984+0000 mgr.smithi067.srbzpc (mgr.14156) 146 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:10.605224+0000 mon.smithi067 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.169:0/922387553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:11.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:11.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 -- 172.21.15.169:0/1544720904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 msgr2=0x7fbee80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 --2- 172.21.15.169:0/1544720904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee80f58d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fbedc009fb0 tx=0x7fbedc009b30).stop 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 -- 172.21.15.169:0/1544720904 shutdown_connections 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 --2- 172.21.15.169:0/1544720904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee80f58d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 -- 172.21.15.169:0/1544720904 >> 172.21.15.169:0/1544720904 conn(0x7fbee80f0c10 msgr2=0x7fbee80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 -- 172.21.15.169:0/1544720904 shutdown_connections 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.127+0000 7fbeeeae0700 1 -- 172.21.15.169:0/1544720904 wait complete. 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.128+0000 7fbeeeae0700 1 Processor -- start 2024-01-11T05:08:13.129 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.128+0000 7fbeeeae0700 1 -- start start 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.128+0000 7fbeeeae0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee81094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.128+0000 7fbeeeae0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee8060300 con 0x7fbee80f54b0 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.128+0000 7fbeec87c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee81094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbeec87c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee81094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45512/0 (socket says 172.21.15.169:45512) 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbeec87c700 1 -- 172.21.15.169:0/3288601021 learned_addr learned my addr 172.21.15.169:0/3288601021 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbeec87c700 1 -- 172.21.15.169:0/3288601021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc005040 con 0x7fbee80f54b0 2024-01-11T05:08:13.131 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbeec87c700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee81094c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fbedc0089f0 tx=0x7fbedc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc009320 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee8060500 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbedc013ec0 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee8109b70 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.129+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc014d30 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.130+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbedc0095f0 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.130+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee80591e0 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.130+0000 7fbee57fa700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed8031090 0x7fbed8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.130+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbedc042440 con 0x7fbee80f54b0 2024-01-11T05:08:13.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.130+0000 7fbee7fff700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed8031090 0x7fbed8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:13.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.131+0000 7fbee7fff700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed8031090 0x7fbed8033550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbed40096a0 tx=0x7fbed4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:13.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.135+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbedc01d7f0 con 0x7fbee80f54b0 2024-01-11T05:08:13.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.346+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbee80fafa0 con 0x7fbee80f54b0 2024-01-11T05:08:13.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.346+0000 7fbee57fa700 1 -- 172.21.15.169:0/3288601021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbedc01d7f0 con 0x7fbee80f54b0 2024-01-11T05:08:13.348 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:13.348 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:13.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed8031090 msgr2=0x7fbed8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:13.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed8031090 0x7fbed8033550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbed40096a0 tx=0x7fbed4006b40).stop 2024-01-11T05:08:13.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 msgr2=0x7fbee81094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:13.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee81094c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fbedc0089f0 tx=0x7fbedc00b040).stop 2024-01-11T05:08:13.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 shutdown_connections 2024-01-11T05:08:13.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbed8031090 0x7fbed8033550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:13.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 --2- 172.21.15.169:0/3288601021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee80f54b0 0x7fbee81094c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:13.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 >> 172.21.15.169:0/3288601021 conn(0x7fbee80f0c10 msgr2=0x7fbee80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:13.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 shutdown_connections 2024-01-11T05:08:13.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:13.348+0000 7fbeeeae0700 1 -- 172.21.15.169:0/3288601021 wait complete. 2024-01-11T05:08:13.350 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:12.315287+0000 mgr.smithi067.srbzpc (mgr.14156) 147 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:13.347956+0000 mon.smithi067 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.169:0/3288601021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:14.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:14.724 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:15.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:14.315620+0000 mgr.smithi067.srbzpc (mgr.14156) 148 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:15.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 -- 172.21.15.169:0/2648661512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 msgr2=0x7f94a00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:15.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 --2- 172.21.15.169:0/2648661512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a00f69b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f949c004760 tx=0x7f949c009ad0).stop 2024-01-11T05:08:15.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 -- 172.21.15.169:0/2648661512 shutdown_connections 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 --2- 172.21.15.169:0/2648661512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a00f69b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 -- 172.21.15.169:0/2648661512 >> 172.21.15.169:0/2648661512 conn(0x7f94a00f1cd0 msgr2=0x7f94a00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 -- 172.21.15.169:0/2648661512 shutdown_connections 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.843+0000 7f94a74a5700 1 -- 172.21.15.169:0/2648661512 wait complete. 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a74a5700 1 Processor -- start 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a74a5700 1 -- start start 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a74a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a74a5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0106cb0 con 0x7f94a00f6590 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a5241700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a5241700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45532/0 (socket says 172.21.15.169:45532) 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.844+0000 7f94a5241700 1 -- 172.21.15.169:0/3541486637 learned_addr learned my addr 172.21.15.169:0/3541486637 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f94a5241700 1 -- 172.21.15.169:0/3541486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c009d70 con 0x7f94a00f6590 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f94a5241700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a01067c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f949c013f10 tx=0x7f949c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f949c00e040 con 0x7f94a00f6590 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a0106eb0 con 0x7f94a00f6590 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f949c007df0 con 0x7f94a00f6590 2024-01-11T05:08:15.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f949c014d60 con 0x7f94a00f6590 2024-01-11T05:08:15.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.845+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a0107340 con 0x7f94a00f6590 2024-01-11T05:08:15.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.846+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f949c014520 con 0x7f94a00f6590 2024-01-11T05:08:15.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.846+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a0045bb0 con 0x7f94a00f6590 2024-01-11T05:08:15.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.846+0000 7f9495ffb700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f948c0310e0 0x7f948c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:15.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.846+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f949c042580 con 0x7f94a00f6590 2024-01-11T05:08:15.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.846+0000 7f94a4a40700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f948c0310e0 0x7f948c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:15.849 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.847+0000 7f94a4a40700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f948c0310e0 0x7f948c0335a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f94900096a0 tx=0x7f9490006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:15.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:15.851+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f949c007960 con 0x7f94a00f6590 2024-01-11T05:08:16.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.062+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94a0108190 con 0x7f94a00f6590 2024-01-11T05:08:16.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.063+0000 7f9495ffb700 1 -- 172.21.15.169:0/3541486637 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f949c0262f0 con 0x7f94a00f6590 2024-01-11T05:08:16.064 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:16.064 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:16.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f948c0310e0 msgr2=0x7f948c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:16.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f948c0310e0 0x7f948c0335a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f94900096a0 tx=0x7f9490006b40).stop 2024-01-11T05:08:16.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 msgr2=0x7f94a01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:16.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a01067c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f949c013f10 tx=0x7f949c009ad0).stop 2024-01-11T05:08:16.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 shutdown_connections 2024-01-11T05:08:16.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f948c0310e0 0x7f948c0335a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:16.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 --2- 172.21.15.169:0/3541486637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a00f6590 0x7f94a01067c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:16.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 >> 172.21.15.169:0/3541486637 conn(0x7f94a00f1cd0 msgr2=0x7f94a00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:16.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 shutdown_connections 2024-01-11T05:08:16.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:16.065+0000 7f94a74a5700 1 -- 172.21.15.169:0/3541486637 wait complete. 2024-01-11T05:08:16.068 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:16.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:16 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:16.064600+0000 mon.smithi067 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.169:0/3541486637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:17.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:17.449 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:17.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:16.315916+0000 mgr.smithi067.srbzpc (mgr.14156) 149 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:18.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 -- 172.21.15.169:0/3663528847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 msgr2=0x7f88300f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 --2- 172.21.15.169:0/3663528847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88300f58a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8824009fb0 tx=0x7f8824009b30).stop 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 -- 172.21.15.169:0/3663528847 shutdown_connections 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 --2- 172.21.15.169:0/3663528847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88300f58a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 -- 172.21.15.169:0/3663528847 >> 172.21.15.169:0/3663528847 conn(0x7f88300f0be0 msgr2=0x7f88300f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 -- 172.21.15.169:0/3663528847 shutdown_connections 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.608+0000 7f88361fc700 1 -- 172.21.15.169:0/3663528847 wait complete. 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f88361fc700 1 Processor -- start 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f88361fc700 1 -- start start 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f88361fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88301056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f88361fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830105be0 con 0x7f88300f5480 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f882f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88301056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:18.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f882f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88301056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45550/0 (socket says 172.21.15.169:45550) 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f882f7fe700 1 -- 172.21.15.169:0/1543071024 learned_addr learned my addr 172.21.15.169:0/1543071024 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.609+0000 7f882f7fe700 1 -- 172.21.15.169:0/1543071024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8824005040 con 0x7f88300f5480 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.610+0000 7f882f7fe700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88301056f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f88240089f0 tx=0x7f882400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.610+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8824009320 con 0x7f88300f5480 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.610+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830105de0 con 0x7f88300f5480 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.610+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8824013ec0 con 0x7f88300f5480 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.610+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8824014d30 con 0x7f88300f5480 2024-01-11T05:08:18.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.610+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8830106270 con 0x7f88300f5480 2024-01-11T05:08:18.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.611+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f88240095f0 con 0x7f88300f5480 2024-01-11T05:08:18.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.611+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8830045bb0 con 0x7f88300f5480 2024-01-11T05:08:18.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.611+0000 7f8817fff700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8818031090 0x7f8818033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:18.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.611+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f88240423e0 con 0x7f88300f5480 2024-01-11T05:08:18.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.611+0000 7f882effd700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8818031090 0x7f8818033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:18.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.612+0000 7f882effd700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8818031090 0x7f8818033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f88200096a0 tx=0x7f8820006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:18.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.616+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f882401d9d0 con 0x7f88300f5480 2024-01-11T05:08:18.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.825+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88300faf70 con 0x7f88300f5480 2024-01-11T05:08:18.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.826+0000 7f8817fff700 1 -- 172.21.15.169:0/1543071024 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f882401dbe0 con 0x7f88300f5480 2024-01-11T05:08:18.827 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:18.827 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8818031090 msgr2=0x7f8818033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8818031090 0x7f8818033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f88200096a0 tx=0x7f8820006b40).stop 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 msgr2=0x7f88301056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88301056f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f88240089f0 tx=0x7f882400b040).stop 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 shutdown_connections 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8818031090 0x7f8818033550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 --2- 172.21.15.169:0/1543071024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88300f5480 0x7f88301056f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 >> 172.21.15.169:0/1543071024 conn(0x7f88300f0be0 msgr2=0x7f88300f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 shutdown_connections 2024-01-11T05:08:18.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:18.828+0000 7f88361fc700 1 -- 172.21.15.169:0/1543071024 wait complete. 2024-01-11T05:08:18.830 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:18.316230+0000 mgr.smithi067.srbzpc (mgr.14156) 150 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:18.827497+0000 mon.smithi067 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.169:0/1543071024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:20.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:20.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:21.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- 172.21.15.169:0/3873947647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 msgr2=0x7fcc900f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:21.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 --2- 172.21.15.169:0/3873947647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc900f56f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fcc8c009fb0 tx=0x7fcc8c009b30).stop 2024-01-11T05:08:21.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- 172.21.15.169:0/3873947647 shutdown_connections 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 --2- 172.21.15.169:0/3873947647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc900f56f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- 172.21.15.169:0/3873947647 >> 172.21.15.169:0/3873947647 conn(0x7fcc900f0a30 msgr2=0x7fcc900f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- 172.21.15.169:0/3873947647 shutdown_connections 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- 172.21.15.169:0/3873947647 wait complete. 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 Processor -- start 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- start start 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc90106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.332+0000 7fcc96927700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90106b10 con 0x7fcc900f52d0 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc95925700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc90106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc95925700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc90106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:58928/0 (socket says 172.21.15.169:58928) 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc95925700 1 -- 172.21.15.169:0/3675009994 learned_addr learned my addr 172.21.15.169:0/3675009994 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:21.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc95925700 1 -- 172.21.15.169:0/3675009994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c005040 con 0x7fcc900f52d0 2024-01-11T05:08:21.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc95925700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc90106620 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fcc8c0089f0 tx=0x7fcc8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:21.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8c009320 con 0x7fcc900f52d0 2024-01-11T05:08:21.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc90106d10 con 0x7fcc900f52d0 2024-01-11T05:08:21.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcc8c013ec0 con 0x7fcc900f52d0 2024-01-11T05:08:21.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8c014d30 con 0x7fcc900f52d0 2024-01-11T05:08:21.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.333+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc901071a0 con 0x7fcc900f52d0 2024-01-11T05:08:21.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.334+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcc8c0095f0 con 0x7fcc900f52d0 2024-01-11T05:08:21.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.334+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc90045bb0 con 0x7fcc900f52d0 2024-01-11T05:08:21.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.334+0000 7fcc867fc700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fcc7c0310e0 0x7fcc7c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:21.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.334+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcc8c0424a0 con 0x7fcc900f52d0 2024-01-11T05:08:21.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.335+0000 7fcc95124700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fcc7c0310e0 0x7fcc7c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:21.337 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.335+0000 7fcc95124700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fcc7c0310e0 0x7fcc7c0335a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fcc800096a0 tx=0x7fcc80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:21.340 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.340+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcc8c01d7f0 con 0x7fcc900f52d0 2024-01-11T05:08:21.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.547+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc90108050 con 0x7fcc900f52d0 2024-01-11T05:08:21.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.548+0000 7fcc867fc700 1 -- 172.21.15.169:0/3675009994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc8c01d7f0 con 0x7fcc900f52d0 2024-01-11T05:08:21.549 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:21.549 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fcc7c0310e0 msgr2=0x7fcc7c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fcc7c0310e0 0x7fcc7c0335a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fcc800096a0 tx=0x7fcc80006b40).stop 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 msgr2=0x7fcc90106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc90106620 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fcc8c0089f0 tx=0x7fcc8c00b040).stop 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 shutdown_connections 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fcc7c0310e0 0x7fcc7c0335a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 --2- 172.21.15.169:0/3675009994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900f52d0 0x7fcc90106620 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 >> 172.21.15.169:0/3675009994 conn(0x7fcc900f0a30 msgr2=0x7fcc9005ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 shutdown_connections 2024-01-11T05:08:21.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:21.550+0000 7fcc96927700 1 -- 172.21.15.169:0/3675009994 wait complete. 2024-01-11T05:08:21.552 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:20.316481+0000 mgr.smithi067.srbzpc (mgr.14156) 151 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:22.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:22 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:21.549399+0000 mon.smithi067 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.169:0/3675009994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:22.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:22.876 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:22.316848+0000 mgr.smithi067.srbzpc (mgr.14156) 152 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:22.416004+0000 mon.smithi067 (mon.0) 217 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:08:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:22.416361+0000 mon.smithi067 (mon.0) 218 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 -- 172.21.15.169:0/1610177270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 msgr2=0x7fdf880f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 --2- 172.21.15.169:0/1610177270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf880f5e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdf7c009fb0 tx=0x7fdf7c009b30).stop 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 -- 172.21.15.169:0/1610177270 shutdown_connections 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 --2- 172.21.15.169:0/1610177270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf880f5e70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 -- 172.21.15.169:0/1610177270 >> 172.21.15.169:0/1610177270 conn(0x7fdf880f0360 msgr2=0x7fdf880f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 -- 172.21.15.169:0/1610177270 shutdown_connections 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.035+0000 7fdf8f123700 1 -- 172.21.15.169:0/1610177270 wait complete. 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8f123700 1 Processor -- start 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8f123700 1 -- start start 2024-01-11T05:08:24.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8f123700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf881067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:24.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8f123700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf88106ce0 con 0x7fdf880f5a50 2024-01-11T05:08:24.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8cebf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf881067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:24.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8cebf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf881067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:58944/0 (socket says 172.21.15.169:58944) 2024-01-11T05:08:24.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.036+0000 7fdf8cebf700 1 -- 172.21.15.169:0/1192689180 learned_addr learned my addr 172.21.15.169:0/1192689180 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:24.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf8cebf700 1 -- 172.21.15.169:0/1192689180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c005040 con 0x7fdf880f5a50 2024-01-11T05:08:24.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf8cebf700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf881067f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fdf7c0089f0 tx=0x7fdf7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:24.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7c009320 con 0x7fdf880f5a50 2024-01-11T05:08:24.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf7c013ec0 con 0x7fdf880f5a50 2024-01-11T05:08:24.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf88106ee0 con 0x7fdf880f5a50 2024-01-11T05:08:24.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7c014d30 con 0x7fdf880f5a50 2024-01-11T05:08:24.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.037+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf88107370 con 0x7fdf880f5a50 2024-01-11T05:08:24.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.038+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf88045bb0 con 0x7fdf880f5a50 2024-01-11T05:08:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.040+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdf7c0095f0 con 0x7fdf880f5a50 2024-01-11T05:08:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.040+0000 7fdf857fa700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdf780310e0 0x7fdf780335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.040+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdf7c0424f0 con 0x7fdf880f5a50 2024-01-11T05:08:24.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.040+0000 7fdf87fff700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdf780310e0 0x7fdf780335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:24.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.040+0000 7fdf87fff700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdf780310e0 0x7fdf780335a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fdf740096a0 tx=0x7fdf74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:24.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.044+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdf7c00f990 con 0x7fdf880f5a50 2024-01-11T05:08:24.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.252+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf881081c0 con 0x7fdf880f5a50 2024-01-11T05:08:24.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.252+0000 7fdf857fa700 1 -- 172.21.15.169:0/1192689180 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf7c026e40 con 0x7fdf880f5a50 2024-01-11T05:08:24.254 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:24.254 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdf780310e0 msgr2=0x7fdf780335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdf780310e0 0x7fdf780335a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fdf740096a0 tx=0x7fdf74006b40).stop 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 msgr2=0x7fdf881067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf881067f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fdf7c0089f0 tx=0x7fdf7c00b040).stop 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 shutdown_connections 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdf780310e0 0x7fdf780335a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:24.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 --2- 172.21.15.169:0/1192689180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf880f5a50 0x7fdf881067f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:24.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 >> 172.21.15.169:0/1192689180 conn(0x7fdf880f0360 msgr2=0x7fdf880f5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:24.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 shutdown_connections 2024-01-11T05:08:24.256 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:24.254+0000 7fdf8f123700 1 -- 172.21.15.169:0/1192689180 wait complete. 2024-01-11T05:08:24.256 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:24.253837+0000 mon.smithi067 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.169:0/1192689180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:24.317181+0000 mgr.smithi067.srbzpc (mgr.14156) 153 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:25.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:25.622 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:26.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 -- 172.21.15.169:0/3183735019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 msgr2=0x7f15840f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:26.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 --2- 172.21.15.169:0/3183735019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15840f58a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1578009fb0 tx=0x7f1578009b30).stop 2024-01-11T05:08:26.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 -- 172.21.15.169:0/3183735019 shutdown_connections 2024-01-11T05:08:26.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 --2- 172.21.15.169:0/3183735019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15840f58a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:26.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 -- 172.21.15.169:0/3183735019 >> 172.21.15.169:0/3183735019 conn(0x7f15840f0be0 msgr2=0x7f15840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:26.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 -- 172.21.15.169:0/3183735019 shutdown_connections 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 -- 172.21.15.169:0/3183735019 wait complete. 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.709+0000 7f1589b4b700 1 Processor -- start 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f1589b4b700 1 -- start start 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f1589b4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15841056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f1589b4b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584105be0 con 0x7f15840f5480 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f15837fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15841056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f15837fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15841056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:58968/0 (socket says 172.21.15.169:58968) 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f15837fe700 1 -- 172.21.15.169:0/2138119388 learned_addr learned my addr 172.21.15.169:0/2138119388 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f15837fe700 1 -- 172.21.15.169:0/2138119388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578005040 con 0x7f15840f5480 2024-01-11T05:08:26.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f15837fe700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15841056f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f15780089f0 tx=0x7f157800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:26.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.710+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1578009320 con 0x7f15840f5480 2024-01-11T05:08:26.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.711+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1584105de0 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.711+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1578013ec0 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.711+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1578014d30 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.711+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1584106270 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.711+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f15780095f0 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.711+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1584045bb0 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.712+0000 7f156bfff700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f156c0310e0 0x7f156c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.712+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f15780424a0 con 0x7f15840f5480 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.712+0000 7f1582ffd700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f156c0310e0 0x7f156c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:26.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.712+0000 7f1582ffd700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f156c0310e0 0x7f156c0335a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f15740096a0 tx=0x7f1574006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:26.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.716+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f157801d980 con 0x7f15840f5480 2024-01-11T05:08:26.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.924+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15840faf70 con 0x7f15840f5480 2024-01-11T05:08:26.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.924+0000 7f156bfff700 1 -- 172.21.15.169:0/2138119388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f157801d380 con 0x7f15840f5480 2024-01-11T05:08:26.925 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:26.926 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:26.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f156c0310e0 msgr2=0x7f156c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:26.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f156c0310e0 0x7f156c0335a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f15740096a0 tx=0x7f1574006b40).stop 2024-01-11T05:08:26.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 msgr2=0x7f15841056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:26.927 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15841056f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f15780089f0 tx=0x7f157800b040).stop 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 shutdown_connections 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f156c0310e0 0x7f156c0335a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 --2- 172.21.15.169:0/2138119388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15840f5480 0x7f15841056f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 >> 172.21.15.169:0/2138119388 conn(0x7f15840f0be0 msgr2=0x7f15840f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 shutdown_connections 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:26.926+0000 7f1589b4b700 1 -- 172.21.15.169:0/2138119388 wait complete. 2024-01-11T05:08:26.928 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:26.317518+0000 mgr.smithi067.srbzpc (mgr.14156) 154 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:26.925825+0000 mon.smithi067 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.169:0/2138119388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:28.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:28.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:29.394 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 -- 172.21.15.169:0/4101306070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 msgr2=0x7fc8100f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 --2- 172.21.15.169:0/4101306070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc8100f9160 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fc800009fb0 tx=0x7fc800009b30).stop 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 -- 172.21.15.169:0/4101306070 shutdown_connections 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 --2- 172.21.15.169:0/4101306070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc8100f9160 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 -- 172.21.15.169:0/4101306070 >> 172.21.15.169:0/4101306070 conn(0x7fc8100f0be0 msgr2=0x7fc8100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 -- 172.21.15.169:0/4101306070 shutdown_connections 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 -- 172.21.15.169:0/4101306070 wait complete. 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.393+0000 7fc816351700 1 Processor -- start 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc816351700 1 -- start start 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc816351700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc810106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:29.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc816351700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc810106c40 con 0x7fc810060360 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc80ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc810106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc80ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc810106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51716/0 (socket says 172.21.15.169:51716) 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc80ffff700 1 -- 172.21.15.169:0/1952413891 learned_addr learned my addr 172.21.15.169:0/1952413891 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.394+0000 7fc80ffff700 1 -- 172.21.15.169:0/1952413891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc800005040 con 0x7fc810060360 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.395+0000 7fc80ffff700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc810106750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc800009f80 tx=0x7fc80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.395+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc800009320 con 0x7fc810060360 2024-01-11T05:08:29.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.395+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc810106e40 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.395+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc800013680 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.395+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80001eda0 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.395+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8101072d0 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.396+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc800014420 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.396+0000 7fc80cff9700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7f8031090 0x7fc7f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.396+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc800042390 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.396+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100591e0 con 0x7fc810060360 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.396+0000 7fc80f7fe700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7f8031090 0x7fc7f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:29.398 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.396+0000 7fc80f7fe700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7f8031090 0x7fc7f8033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc8040096a0 tx=0x7fc804006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:29.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.401+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc800014700 con 0x7fc810060360 2024-01-11T05:08:29.609 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.608+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc810045bb0 con 0x7fc810060360 2024-01-11T05:08:29.610 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.609+0000 7fc80cff9700 1 -- 172.21.15.169:0/1952413891 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc80001c320 con 0x7fc810060360 2024-01-11T05:08:29.610 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:29.610 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:29.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7f8031090 msgr2=0x7fc7f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:29.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7f8031090 0x7fc7f8033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc8040096a0 tx=0x7fc804006b40).stop 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 msgr2=0x7fc810106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc810106750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc800009f80 tx=0x7fc80000b040).stop 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 shutdown_connections 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc7f8031090 0x7fc7f8033550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 --2- 172.21.15.169:0/1952413891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc810060360 0x7fc810106750 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.610+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 >> 172.21.15.169:0/1952413891 conn(0x7fc8100f0be0 msgr2=0x7fc8100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.611+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 shutdown_connections 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:29.611+0000 7fc816351700 1 -- 172.21.15.169:0/1952413891 wait complete. 2024-01-11T05:08:29.612 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:28.317820+0000 mgr.smithi067.srbzpc (mgr.14156) 155 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:30.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:30 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:29.610122+0000 mon.smithi067 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.169:0/1952413891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:30.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:30.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:30.318124+0000 mgr.smithi067.srbzpc (mgr.14156) 156 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:32.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 -- 172.21.15.169:0/3598223804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 msgr2=0x7f2a540f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:32.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 --2- 172.21.15.169:0/3598223804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a540f56f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2a48009fb0 tx=0x7f2a48009b30).stop 2024-01-11T05:08:32.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 -- 172.21.15.169:0/3598223804 shutdown_connections 2024-01-11T05:08:32.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 --2- 172.21.15.169:0/3598223804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a540f56f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:32.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 -- 172.21.15.169:0/3598223804 >> 172.21.15.169:0/3598223804 conn(0x7f2a540f0a30 msgr2=0x7f2a540f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:32.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 -- 172.21.15.169:0/3598223804 shutdown_connections 2024-01-11T05:08:32.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 -- 172.21.15.169:0/3598223804 wait complete. 2024-01-11T05:08:32.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 Processor -- start 2024-01-11T05:08:32.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.099+0000 7f2a598d2700 1 -- start start 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a598d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a54109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a598d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54060320 con 0x7f2a540f52d0 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a588d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a54109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a588d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a54109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51740/0 (socket says 172.21.15.169:51740) 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a588d0700 1 -- 172.21.15.169:0/944032842 learned_addr learned my addr 172.21.15.169:0/944032842 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a588d0700 1 -- 172.21.15.169:0/944032842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a48005040 con 0x7f2a540f52d0 2024-01-11T05:08:32.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.100+0000 7f2a588d0700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a54109310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2a480089f0 tx=0x7f2a4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:32.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.101+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a48009320 con 0x7f2a540f52d0 2024-01-11T05:08:32.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.101+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a54060520 con 0x7f2a540f52d0 2024-01-11T05:08:32.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.101+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2a48013ec0 con 0x7f2a540f52d0 2024-01-11T05:08:32.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.101+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a48014d30 con 0x7f2a540f52d0 2024-01-11T05:08:32.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.101+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a541099c0 con 0x7f2a540f52d0 2024-01-11T05:08:32.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.102+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2a480095f0 con 0x7f2a540f52d0 2024-01-11T05:08:32.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.102+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a54045bb0 con 0x7f2a540f52d0 2024-01-11T05:08:32.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.102+0000 7f2a517fa700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a44031090 0x7f2a44033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:32.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.102+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2a480423e0 con 0x7f2a540f52d0 2024-01-11T05:08:32.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.102+0000 7f2a53fff700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a44031090 0x7f2a44033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:32.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.102+0000 7f2a53fff700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a44031090 0x7f2a44033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2a400096a0 tx=0x7f2a40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:32.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.107+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2a4801d7f0 con 0x7f2a540f52d0 2024-01-11T05:08:32.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.316+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a5410a1e0 con 0x7f2a540f52d0 2024-01-11T05:08:32.317 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.317+0000 7f2a517fa700 1 -- 172.21.15.169:0/944032842 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a4801d7f0 con 0x7f2a540f52d0 2024-01-11T05:08:32.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:32.318 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:32.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a44031090 msgr2=0x7f2a44033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:32.319 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a44031090 0x7f2a44033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2a400096a0 tx=0x7f2a40006b40).stop 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 msgr2=0x7f2a54109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a54109310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2a480089f0 tx=0x7f2a4800b040).stop 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 shutdown_connections 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a44031090 0x7f2a44033550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 --2- 172.21.15.169:0/944032842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a540f52d0 0x7f2a54109310 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.318+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 >> 172.21.15.169:0/944032842 conn(0x7f2a540f0a30 msgr2=0x7f2a540f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.319+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 shutdown_connections 2024-01-11T05:08:32.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:32.319+0000 7f2a598d2700 1 -- 172.21.15.169:0/944032842 wait complete. 2024-01-11T05:08:32.321 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:32.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:32 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:32.317920+0000 mon.smithi067 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.169:0/944032842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:33.666 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:32.318331+0000 mgr.smithi067.srbzpc (mgr.14156) 157 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:33.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:33.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:34.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:08:33.666058+0000 mgr.smithi067.srbzpc (mgr.14156) 158 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-89773ea0-b03e-11ee-95ab-87774f69a715-alertmanager.smithi067 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: /bin/podman: stderr Error: no such container ceph-89773ea0-b03e-11ee-95ab-87774f69a715-alertmanager.smithi067 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Deploy daemon alertmanager.smithi067 ... 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi067 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:04:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:04:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:05:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-11T05:08:34.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi067 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:06:40Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:07:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:07:56Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: main() 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-01-11T05:08:34.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: r = ctx.func(ctx) 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: return func(ctx) 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: raise RuntimeError('uid/gid not found') 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: RuntimeError: uid/gid not found 2024-01-11T05:08:34.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: yield (conn, connr) 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: code, '\n'.join(err))) 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-89773ea0-b03e-11ee-95ab-87774f69a715-alertmanager.smithi067 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: /bin/podman: stderr Error: no such container ceph-89773ea0-b03e-11ee-95ab-87774f69a715-alertmanager.smithi067 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Deploy daemon alertmanager.smithi067 ... 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi067 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:04:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:04:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:05:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi067 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:06:40Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:07:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:07:56Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: main() 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: r = ctx.func(ctx) 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: return func(ctx) 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-01-11T05:08:34.711 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: raise RuntimeError('uid/gid not found') 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: RuntimeError: uid/gid not found 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:08:33.666370+0000 mgr.smithi067.srbzpc (mgr.14156) 159 : cephadm [ERR] Failed while placing alertmanager.smithi067 on smithi067: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-89773ea0-b03e-11ee-95ab-87774f69a715-alertmanager.smithi067 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: /bin/podman: stderr Error: no such container ceph-89773ea0-b03e-11ee-95ab-87774f69a715-alertmanager.smithi067 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Deploy daemon alertmanager.smithi067 ... 2024-01-11T05:08:34.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi067 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:04:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:04:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:05:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi067 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:06:40Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:07:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.713 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr time="2024-01-11T05:07:56Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: main() 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: r = ctx.func(ctx) 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: return func(ctx) 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-11T05:08:34.714 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: raise RuntimeError('uid/gid not found') 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: RuntimeError: uid/gid not found 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:33.668153+0000 mon.smithi067 (mon.0) 223 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:33.670859+0000 mon.smithi067 (mon.0) 224 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:33.671723+0000 mon.smithi067 (mon.0) 225 : audit [DBG] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-11T05:08:34.715 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:08:33.672355+0000 mgr.smithi067.srbzpc (mgr.14156) 160 : cephadm [INF] Deploying daemon crash.smithi067 on smithi067 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 -- 172.21.15.169:0/1451775418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 msgr2=0x7f19340f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 --2- 172.21.15.169:0/1451775418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19340f58a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f1928009fb0 tx=0x7f1928009b30).stop 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 -- 172.21.15.169:0/1451775418 shutdown_connections 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 --2- 172.21.15.169:0/1451775418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19340f58a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 -- 172.21.15.169:0/1451775418 >> 172.21.15.169:0/1451775418 conn(0x7f19340f0be0 msgr2=0x7f19340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 -- 172.21.15.169:0/1451775418 shutdown_connections 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 -- 172.21.15.169:0/1451775418 wait complete. 2024-01-11T05:08:34.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.822+0000 7f1939ffb700 1 Processor -- start 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f1939ffb700 1 -- start start 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f1939ffb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f1939ffb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934106cd0 con 0x7f19340f5480 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f19337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f19337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51752/0 (socket says 172.21.15.169:51752) 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f19337fe700 1 -- 172.21.15.169:0/4116008515 learned_addr learned my addr 172.21.15.169:0/4116008515 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f19337fe700 1 -- 172.21.15.169:0/4116008515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928005040 con 0x7f19340f5480 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.823+0000 7f19337fe700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19341067e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f19280089f0 tx=0x7f192800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:34.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.824+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1928009320 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.824+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1934106ed0 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.824+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1928013ec0 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.824+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1928014d30 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.824+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1934107360 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.824+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1928014420 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.825+0000 7f191bfff700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f191c031090 0x7f191c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.825+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f19280424a0 con 0x7f19340f5480 2024-01-11T05:08:34.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.825+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1934045bb0 con 0x7f19340f5480 2024-01-11T05:08:34.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.825+0000 7f1932ffd700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f191c031090 0x7f191c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:34.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.825+0000 7f1932ffd700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f191c031090 0x7f191c033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f19240096a0 tx=0x7f1924006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:34.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:34.830+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1928014750 con 0x7f19340f5480 2024-01-11T05:08:35.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.039+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1934108080 con 0x7f19340f5480 2024-01-11T05:08:35.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.039+0000 7f191bfff700 1 -- 172.21.15.169:0/4116008515 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f192801d7f0 con 0x7f19340f5480 2024-01-11T05:08:35.040 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:35.041 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:35.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f191c031090 msgr2=0x7f191c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:35.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f191c031090 0x7f191c033550 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f19240096a0 tx=0x7f1924006b40).stop 2024-01-11T05:08:35.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 msgr2=0x7f19341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:35.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19341067e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f19280089f0 tx=0x7f192800b040).stop 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 shutdown_connections 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f191c031090 0x7f191c033550 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 --2- 172.21.15.169:0/4116008515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19340f5480 0x7f19341067e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 >> 172.21.15.169:0/4116008515 conn(0x7f19340f0be0 msgr2=0x7f193405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 shutdown_connections 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:35.041+0000 7f1939ffb700 1 -- 172.21.15.169:0/4116008515 wait complete. 2024-01-11T05:08:35.043 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:34.318705+0000 mgr.smithi067.srbzpc (mgr.14156) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.040639+0000 mon.smithi067 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.169:0/4116008515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:35.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.195708+0000 mon.smithi067 (mon.0) 227 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:35.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.255415+0000 mon.smithi067 (mon.0) 228 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:35.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.257444+0000 mon.smithi067 (mon.0) 229 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:35.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.257827+0000 mon.smithi067 (mon.0) 230 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-11T05:08:35.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.260422+0000 mon.smithi067 (mon.0) 231 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:36.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:36.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:36.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:35.258148+0000 mgr.smithi067.srbzpc (mgr.14156) 162 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-11T05:08:36.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:36 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:08:35.263297+0000 mgr.smithi067.srbzpc (mgr.14156) 163 : cephadm [INF] Deploying daemon grafana.smithi067 on smithi067 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- 172.21.15.169:0/2288777284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 msgr2=0x7f4e540f4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 --2- 172.21.15.169:0/2288777284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e540f4ab0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4e44009fb0 tx=0x7f4e44009b30).stop 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- 172.21.15.169:0/2288777284 shutdown_connections 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 --2- 172.21.15.169:0/2288777284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e540f4ab0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- 172.21.15.169:0/2288777284 >> 172.21.15.169:0/2288777284 conn(0x7f4e540f0190 msgr2=0x7f4e540f25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- 172.21.15.169:0/2288777284 shutdown_connections 2024-01-11T05:08:37.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- 172.21.15.169:0/2288777284 wait complete. 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 Processor -- start 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- start start 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e54106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.501+0000 7f4e53fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54106b00 con 0x7f4e540f4690 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e54106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e52ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e54106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51776/0 (socket says 172.21.15.169:51776) 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e52ffd700 1 -- 172.21.15.169:0/1382863991 learned_addr learned my addr 172.21.15.169:0/1382863991 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e52ffd700 1 -- 172.21.15.169:0/1382863991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44005040 con 0x7f4e540f4690 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e52ffd700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e54106610 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4e440089f0 tx=0x7f4e4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e44009320 con 0x7f4e540f4690 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.502+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4e44013ec0 con 0x7f4e540f4690 2024-01-11T05:08:37.504 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.503+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e54106d00 con 0x7f4e540f4690 2024-01-11T05:08:37.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.503+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e44014d30 con 0x7f4e540f4690 2024-01-11T05:08:37.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.503+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e54107190 con 0x7f4e540f4690 2024-01-11T05:08:37.505 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.503+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4e440095f0 con 0x7f4e540f4690 2024-01-11T05:08:37.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.503+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e54045bb0 con 0x7f4e540f4690 2024-01-11T05:08:37.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.503+0000 7f4e3b7fe700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e3c0310e0 0x7f4e3c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:37.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.504+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4e44042420 con 0x7f4e540f4690 2024-01-11T05:08:37.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.504+0000 7f4e527fc700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e3c0310e0 0x7f4e3c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:37.506 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.504+0000 7f4e527fc700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e3c0310e0 0x7f4e3c0335a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f4e480096a0 tx=0x7f4e48006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:37.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.509+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4e4401d7f0 con 0x7f4e540f4690 2024-01-11T05:08:37.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.715+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e54108040 con 0x7f4e540f4690 2024-01-11T05:08:37.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.716+0000 7f4e3b7fe700 1 -- 172.21.15.169:0/1382863991 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e4401d7f0 con 0x7f4e540f4690 2024-01-11T05:08:37.717 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:37.717 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:37.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e3c0310e0 msgr2=0x7f4e3c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:37.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e3c0310e0 0x7f4e3c0335a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f4e480096a0 tx=0x7f4e48006b40).stop 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 msgr2=0x7f4e54106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e54106610 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4e440089f0 tx=0x7f4e4400b040).stop 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 shutdown_connections 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e3c0310e0 0x7f4e3c0335a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 --2- 172.21.15.169:0/1382863991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e540f4690 0x7f4e54106610 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 >> 172.21.15.169:0/1382863991 conn(0x7f4e540f0190 msgr2=0x7f4e540f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 shutdown_connections 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:37.717+0000 7f4e53fff700 1 -- 172.21.15.169:0/1382863991 wait complete. 2024-01-11T05:08:37.719 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:36.319096+0000 mgr.smithi067.srbzpc (mgr.14156) 164 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:38.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:37.413168+0000 mon.smithi067 (mon.0) 232 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:38.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:38 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:37.717134+0000 mon.smithi067 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.169:0/1382863991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:39.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:39.092 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:38.319436+0000 mgr.smithi067.srbzpc (mgr.14156) 165 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:40.174 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 -- 172.21.15.169:0/3710941849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 msgr2=0x7fbea40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 --2- 172.21.15.169:0/3710941849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea40f58a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fbea0009fb0 tx=0x7fbea0009b30).stop 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 -- 172.21.15.169:0/3710941849 shutdown_connections 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 --2- 172.21.15.169:0/3710941849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea40f58a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 -- 172.21.15.169:0/3710941849 >> 172.21.15.169:0/3710941849 conn(0x7fbea40f0be0 msgr2=0x7fbea40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 -- 172.21.15.169:0/3710941849 shutdown_connections 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.172+0000 7fbeab791700 1 -- 172.21.15.169:0/3710941849 wait complete. 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbeab791700 1 Processor -- start 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbeab791700 1 -- start start 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbeab791700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbeab791700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea4060320 con 0x7fbea40f5480 2024-01-11T05:08:40.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbea952d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbea952d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35620/0 (socket says 172.21.15.169:35620) 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbea952d700 1 -- 172.21.15.169:0/941147719 learned_addr learned my addr 172.21.15.169:0/941147719 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbea952d700 1 -- 172.21.15.169:0/941147719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0005040 con 0x7fbea40f5480 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.173+0000 7fbea952d700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea41094f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fbea00089f0 tx=0x7fbea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.174+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbea0009320 con 0x7fbea40f5480 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.174+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbea0013ec0 con 0x7fbea40f5480 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.174+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbea0014d30 con 0x7fbea40f5480 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.174+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea4060520 con 0x7fbea40f5480 2024-01-11T05:08:40.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.174+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea4109ba0 con 0x7fbea40f5480 2024-01-11T05:08:40.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.175+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbea00095f0 con 0x7fbea40f5480 2024-01-11T05:08:40.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.175+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea4045bb0 con 0x7fbea40f5480 2024-01-11T05:08:40.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.175+0000 7fbe99ffb700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe90031090 0x7fbe90033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:40.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.175+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbea00424a0 con 0x7fbea40f5480 2024-01-11T05:08:40.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.175+0000 7fbea8d2c700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe90031090 0x7fbe90033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:40.177 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.175+0000 7fbea8d2c700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe90031090 0x7fbe90033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fbe940096a0 tx=0x7fbe94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:40.180 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.180+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbea001d7f0 con 0x7fbea40f5480 2024-01-11T05:08:40.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.387+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbea40faf70 con 0x7fbea40f5480 2024-01-11T05:08:40.388 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.387+0000 7fbe99ffb700 1 -- 172.21.15.169:0/941147719 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbea001d7f0 con 0x7fbea40f5480 2024-01-11T05:08:40.388 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:40.388 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe90031090 msgr2=0x7fbe90033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe90031090 0x7fbe90033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fbe940096a0 tx=0x7fbe94006b40).stop 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 msgr2=0x7fbea41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea41094f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fbea00089f0 tx=0x7fbea000b040).stop 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 shutdown_connections 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbe90031090 0x7fbe90033550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:40.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 --2- 172.21.15.169:0/941147719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbea40f5480 0x7fbea41094f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:40.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 >> 172.21.15.169:0/941147719 conn(0x7fbea40f0be0 msgr2=0x7fbea40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:40.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 shutdown_connections 2024-01-11T05:08:40.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:40.389+0000 7fbeab791700 1 -- 172.21.15.169:0/941147719 wait complete. 2024-01-11T05:08:40.391 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:40.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:40 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:40.388728+0000 mon.smithi067 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.169:0/941147719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:40.319728+0000 mgr.smithi067.srbzpc (mgr.14156) 166 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:41.817 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:41.818 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:42.916 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 -- 172.21.15.169:0/3986797784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 msgr2=0x7f1e300f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 --2- 172.21.15.169:0/3986797784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e300f58a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1e20009fb0 tx=0x7f1e20009b30).stop 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 -- 172.21.15.169:0/3986797784 shutdown_connections 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 --2- 172.21.15.169:0/3986797784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e300f58a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 -- 172.21.15.169:0/3986797784 >> 172.21.15.169:0/3986797784 conn(0x7f1e300f0be0 msgr2=0x7f1e300f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 -- 172.21.15.169:0/3986797784 shutdown_connections 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.915+0000 7f1e385f4700 1 -- 172.21.15.169:0/3986797784 wait complete. 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e385f4700 1 Processor -- start 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e385f4700 1 -- start start 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e385f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e301089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:42.917 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e385f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30108ed0 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e36390700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e301089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e36390700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e301089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35642/0 (socket says 172.21.15.169:35642) 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e36390700 1 -- 172.21.15.169:0/3900962533 learned_addr learned my addr 172.21.15.169:0/3900962533 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e36390700 1 -- 172.21.15.169:0/3900962533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e20005040 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.916+0000 7f1e36390700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e301089e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1e200089f0 tx=0x7f1e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.917+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e20009320 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.917+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e301090d0 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.917+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1e20013ec0 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.917+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e20014d30 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.917+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e30109560 con 0x7f1e300f5480 2024-01-11T05:08:42.919 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.918+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1e20014420 con 0x7f1e300f5480 2024-01-11T05:08:42.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.918+0000 7f1e26ffd700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1e1c031090 0x7f1e1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:42.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.918+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e30045bb0 con 0x7f1e300f5480 2024-01-11T05:08:42.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.918+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1e200423e0 con 0x7f1e300f5480 2024-01-11T05:08:42.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.918+0000 7f1e35b8f700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1e1c031090 0x7f1e1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:42.920 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.918+0000 7f1e35b8f700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1e1c031090 0x7f1e1c033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1e2c0096a0 tx=0x7f1e2c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:42.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:42.922+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1e2001d7f0 con 0x7f1e300f5480 2024-01-11T05:08:43.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.134+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e3010a410 con 0x7f1e300f5480 2024-01-11T05:08:43.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.134+0000 7f1e26ffd700 1 -- 172.21.15.169:0/3900962533 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e20023250 con 0x7f1e300f5480 2024-01-11T05:08:43.135 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:43.135 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1e1c031090 msgr2=0x7f1e1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1e1c031090 0x7f1e1c033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1e2c0096a0 tx=0x7f1e2c006b40).stop 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 msgr2=0x7f1e301089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e301089e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1e200089f0 tx=0x7f1e2000b040).stop 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 shutdown_connections 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1e1c031090 0x7f1e1c033550 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 --2- 172.21.15.169:0/3900962533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e300f5480 0x7f1e301089e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 >> 172.21.15.169:0/3900962533 conn(0x7f1e300f0be0 msgr2=0x7f1e3005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 shutdown_connections 2024-01-11T05:08:43.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:43.136+0000 7f1e385f4700 1 -- 172.21.15.169:0/3900962533 wait complete. 2024-01-11T05:08:43.138 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:42.320023+0000 mgr.smithi067.srbzpc (mgr.14156) 167 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:43.135524+0000 mon.smithi067 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.169:0/3900962533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:44.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:44.532 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:44.320257+0000 mgr.smithi067.srbzpc (mgr.14156) 168 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:45.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 -- 172.21.15.169:0/1768094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 msgr2=0x7f9f440f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 --2- 172.21.15.169:0/1768094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440f5e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9f34009fb0 tx=0x7f9f34009b30).stop 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 -- 172.21.15.169:0/1768094166 shutdown_connections 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 --2- 172.21.15.169:0/1768094166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440f5e70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 -- 172.21.15.169:0/1768094166 >> 172.21.15.169:0/1768094166 conn(0x7f9f440f0360 msgr2=0x7f9f440f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 -- 172.21.15.169:0/1768094166 shutdown_connections 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.703+0000 7f9f4b896700 1 -- 172.21.15.169:0/1768094166 wait complete. 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f4b896700 1 Processor -- start 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f4b896700 1 -- start start 2024-01-11T05:08:45.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f4b896700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f4b896700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f440fb8e0 con 0x7f9f440f5a50 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f49632700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f49632700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35654/0 (socket says 172.21.15.169:35654) 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f49632700 1 -- 172.21.15.169:0/3660971281 learned_addr learned my addr 172.21.15.169:0/3660971281 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f49632700 1 -- 172.21.15.169:0/3660971281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f34005040 con 0x7f9f440f5a50 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f49632700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440fd460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9f340089f0 tx=0x7f9f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f34009320 con 0x7f9f440f5a50 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.704+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9f34013ec0 con 0x7f9f440f5a50 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.705+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f440fba80 con 0x7f9f440f5a50 2024-01-11T05:08:45.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.705+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f34014d30 con 0x7f9f440f5a50 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.705+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f440fbf10 con 0x7f9f440f5a50 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.705+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9f340095f0 con 0x7f9f440f5a50 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.706+0000 7f9f39ffb700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f300310e0 0x7f9f300335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.706+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9f34042460 con 0x7f9f440f5a50 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.706+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f440591e0 con 0x7f9f440f5a50 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.706+0000 7f9f48e31700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f300310e0 0x7f9f300335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:45.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.706+0000 7f9f48e31700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f300310e0 0x7f9f300335a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9f400096a0 tx=0x7f9f40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:45.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.713+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9f3401d7f0 con 0x7f9f440f5a50 2024-01-11T05:08:45.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.921+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f440faf70 con 0x7f9f440f5a50 2024-01-11T05:08:45.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.922+0000 7f9f39ffb700 1 -- 172.21.15.169:0/3660971281 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f3401d7f0 con 0x7f9f440f5a50 2024-01-11T05:08:45.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:45.923 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:45.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f300310e0 msgr2=0x7f9f300335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:45.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f300310e0 0x7f9f300335a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9f400096a0 tx=0x7f9f40006b40).stop 2024-01-11T05:08:45.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 msgr2=0x7f9f440fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440fd460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9f340089f0 tx=0x7f9f3400b040).stop 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 shutdown_connections 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9f300310e0 0x7f9f300335a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 --2- 172.21.15.169:0/3660971281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f440f5a50 0x7f9f440fd460 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 >> 172.21.15.169:0/3660971281 conn(0x7f9f440f0360 msgr2=0x7f9f440f4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 shutdown_connections 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:45.923+0000 7f9f4b896700 1 -- 172.21.15.169:0/3660971281 wait complete. 2024-01-11T05:08:45.925 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:45.923041+0000 mon.smithi067 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.169:0/3660971281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:47.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:47.281 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:47.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:46.320473+0000 mgr.smithi067.srbzpc (mgr.14156) 169 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 -- 172.21.15.169:0/2742540939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 msgr2=0x7f26e00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 --2- 172.21.15.169:0/2742540939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e00f69b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f26d0009fb0 tx=0x7f26d0009b30).stop 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 -- 172.21.15.169:0/2742540939 shutdown_connections 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 --2- 172.21.15.169:0/2742540939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e00f69b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 -- 172.21.15.169:0/2742540939 >> 172.21.15.169:0/2742540939 conn(0x7f26e00f1cf0 msgr2=0x7f26e00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 -- 172.21.15.169:0/2742540939 shutdown_connections 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.398+0000 7f26e7b6f700 1 -- 172.21.15.169:0/2742540939 wait complete. 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e7b6f700 1 Processor -- start 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e7b6f700 1 -- start start 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e7b6f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:48.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e7b6f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0069f00 con 0x7f26e00f6590 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e590b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e590b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35666/0 (socket says 172.21.15.169:35666) 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e590b700 1 -- 172.21.15.169:0/3247175956 learned_addr learned my addr 172.21.15.169:0/3247175956 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e590b700 1 -- 172.21.15.169:0/3247175956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d0005040 con 0x7f26e00f6590 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.399+0000 7f26e590b700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e006ba80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f26d00089f0 tx=0x7f26d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.400+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d0009320 con 0x7f26e00f6590 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.400+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f26d0013ec0 con 0x7f26e00f6590 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.400+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d0014d30 con 0x7f26e00f6590 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.400+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e006a0a0 con 0x7f26e00f6590 2024-01-11T05:08:48.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.400+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e006a530 con 0x7f26e00f6590 2024-01-11T05:08:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.401+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f26d00263a0 con 0x7f26e00f6590 2024-01-11T05:08:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.401+0000 7f26d67fc700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f26cc0310e0 0x7f26cc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.401+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f26d0042350 con 0x7f26e00f6590 2024-01-11T05:08:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.401+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e00591e0 con 0x7f26e00f6590 2024-01-11T05:08:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.401+0000 7f26e510a700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f26cc0310e0 0x7f26cc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:48.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.401+0000 7f26e510a700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f26cc0310e0 0x7f26cc0335a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f26dc0096a0 tx=0x7f26dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:48.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.406+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f26d00266d0 con 0x7f26e00f6590 2024-01-11T05:08:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.615+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26e0045bb0 con 0x7f26e00f6590 2024-01-11T05:08:48.616 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.615+0000 7f26d67fc700 1 -- 172.21.15.169:0/3247175956 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f26d0012330 con 0x7f26e00f6590 2024-01-11T05:08:48.616 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:48.616 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f26cc0310e0 msgr2=0x7f26cc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f26cc0310e0 0x7f26cc0335a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f26dc0096a0 tx=0x7f26dc006b40).stop 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 msgr2=0x7f26e006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e006ba80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f26d00089f0 tx=0x7f26d000b040).stop 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 shutdown_connections 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f26cc0310e0 0x7f26cc0335a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 --2- 172.21.15.169:0/3247175956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e00f6590 0x7f26e006ba80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 >> 172.21.15.169:0/3247175956 conn(0x7f26e00f1cf0 msgr2=0x7f26e00f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 shutdown_connections 2024-01-11T05:08:48.618 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:48.617+0000 7f26e7b6f700 1 -- 172.21.15.169:0/3247175956 wait complete. 2024-01-11T05:08:48.619 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:48.320678+0000 mgr.smithi067.srbzpc (mgr.14156) 170 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:48.616586+0000 mon.smithi067 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.169:0/3247175956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:49.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:49.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:51.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.077+0000 7f0af4085700 1 -- 172.21.15.169:0/331024572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 msgr2=0x7f0aec0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:51.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.077+0000 7f0af4085700 1 --2- 172.21.15.169:0/331024572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec0f69b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0ae8004760 tx=0x7f0ae8009ad0).stop 2024-01-11T05:08:51.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 -- 172.21.15.169:0/331024572 shutdown_connections 2024-01-11T05:08:51.079 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 --2- 172.21.15.169:0/331024572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec0f69b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 -- 172.21.15.169:0/331024572 >> 172.21.15.169:0/331024572 conn(0x7f0aec0f1cd0 msgr2=0x7f0aec0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 -- 172.21.15.169:0/331024572 shutdown_connections 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 -- 172.21.15.169:0/331024572 wait complete. 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 Processor -- start 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 -- start start 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af4085700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec106cb0 con 0x7f0aec0f6590 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af1e21700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af1e21700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60146/0 (socket says 172.21.15.169:60146) 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.078+0000 7f0af1e21700 1 -- 172.21.15.169:0/3595856639 learned_addr learned my addr 172.21.15.169:0/3595856639 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0af1e21700 1 -- 172.21.15.169:0/3595856639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae8009d70 con 0x7f0aec0f6590 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0af1e21700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec1067c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f0ae8004170 tx=0x7f0ae8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ae800e040 con 0x7f0aec0f6590 2024-01-11T05:08:51.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0ae8008ef0 con 0x7f0aec0f6590 2024-01-11T05:08:51.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec106eb0 con 0x7f0aec0f6590 2024-01-11T05:08:51.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ae8014c90 con 0x7f0aec0f6590 2024-01-11T05:08:51.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.079+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec107340 con 0x7f0aec0f6590 2024-01-11T05:08:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.080+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0ae80283f0 con 0x7f0aec0f6590 2024-01-11T05:08:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.080+0000 7f0ae27fc700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0ad8031090 0x7f0ad8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.080+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0ae8042220 con 0x7f0aec0f6590 2024-01-11T05:08:51.082 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.080+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aec045bb0 con 0x7f0aec0f6590 2024-01-11T05:08:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.080+0000 7f0af1620700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0ad8031090 0x7f0ad8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:51.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.080+0000 7f0af1620700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0ad8031090 0x7f0ad8033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0adc0096a0 tx=0x7f0adc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:51.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.085+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0ae80268f0 con 0x7f0aec0f6590 2024-01-11T05:08:51.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.294+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0aec0fc170 con 0x7f0aec0f6590 2024-01-11T05:08:51.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.295+0000 7f0ae27fc700 1 -- 172.21.15.169:0/3595856639 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ae8026300 con 0x7f0aec0f6590 2024-01-11T05:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:51.295 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.296+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0ad8031090 msgr2=0x7f0ad8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.296+0000 7f0af4085700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0ad8031090 0x7f0ad8033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0adc0096a0 tx=0x7f0adc006b40).stop 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.296+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 msgr2=0x7f0aec1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.296+0000 7f0af4085700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec1067c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f0ae8004170 tx=0x7f0ae8009ad0).stop 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.296+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 shutdown_connections 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.296+0000 7f0af4085700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0ad8031090 0x7f0ad8033550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.297+0000 7f0af4085700 1 --2- 172.21.15.169:0/3595856639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aec0f6590 0x7f0aec1067c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.297+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 >> 172.21.15.169:0/3595856639 conn(0x7f0aec0f1cd0 msgr2=0x7f0aec0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.297+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 shutdown_connections 2024-01-11T05:08:51.297 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:51.297+0000 7f0af4085700 1 -- 172.21.15.169:0/3595856639 wait complete. 2024-01-11T05:08:51.298 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:51.488 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:50.320917+0000 mgr.smithi067.srbzpc (mgr.14156) 171 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:51.488 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:51.295027+0000 mon.smithi067 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.169:0/3595856639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:52.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:52.691 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:53.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:51.978374+0000 mon.smithi067 (mon.0) 239 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:53.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:08:51.980527+0000 mgr.smithi067.srbzpc (mgr.14156) 172 : cephadm [INF] Deploying daemon node-exporter.smithi067 on smithi067 2024-01-11T05:08:53.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:52.321126+0000 mgr.smithi067.srbzpc (mgr.14156) 173 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:53.263 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:52 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:52.416116+0000 mon.smithi067 (mon.0) 240 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 -- 172.21.15.169:0/667012488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 msgr2=0x7ff31c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 --2- 172.21.15.169:0/667012488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c0f56f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7ff30c009fb0 tx=0x7ff30c009b30).stop 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 -- 172.21.15.169:0/667012488 shutdown_connections 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 --2- 172.21.15.169:0/667012488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c0f56f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 -- 172.21.15.169:0/667012488 >> 172.21.15.169:0/667012488 conn(0x7ff31c0f0a30 msgr2=0x7ff31c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 -- 172.21.15.169:0/667012488 shutdown_connections 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 -- 172.21.15.169:0/667012488 wait complete. 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.806+0000 7ff321517700 1 Processor -- start 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff321517700 1 -- start start 2024-01-11T05:08:53.807 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff321517700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:53.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff321517700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c106b30 con 0x7ff31c0f52d0 2024-01-11T05:08:53.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:53.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60162/0 (socket says 172.21.15.169:60162) 2024-01-11T05:08:53.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff31bfff700 1 -- 172.21.15.169:0/697666387 learned_addr learned my addr 172.21.15.169:0/697666387 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:53.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff31bfff700 1 -- 172.21.15.169:0/697666387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c005040 con 0x7ff31c0f52d0 2024-01-11T05:08:53.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.807+0000 7ff31bfff700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c106640 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7ff30c0089f0 tx=0x7ff30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:53.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.808+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30c009320 con 0x7ff31c0f52d0 2024-01-11T05:08:53.809 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.808+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c106d30 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.808+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c1071c0 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.808+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff30c013ec0 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.808+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30c014d30 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.809+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff30c0095f0 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.809+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c045bb0 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.809+0000 7ff318ff9700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff304031090 0x7ff304033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.809+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff30c0424c0 con 0x7ff31c0f52d0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.809+0000 7ff31b7fe700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff304031090 0x7ff304033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:53.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.809+0000 7ff31b7fe700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff304031090 0x7ff304033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff3100096a0 tx=0x7ff310006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:53.813 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:53.814+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff30c01d7f0 con 0x7ff31c0f52d0 2024-01-11T05:08:54.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.020+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff31c0fadc0 con 0x7ff31c0f52d0 2024-01-11T05:08:54.021 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.021+0000 7ff318ff9700 1 -- 172.21.15.169:0/697666387 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff30c00fe40 con 0x7ff31c0f52d0 2024-01-11T05:08:54.021 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:54.021 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff304031090 msgr2=0x7ff304033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff304031090 0x7ff304033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff3100096a0 tx=0x7ff310006b40).stop 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 msgr2=0x7ff31c106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c106640 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7ff30c0089f0 tx=0x7ff30c00b040).stop 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 shutdown_connections 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff304031090 0x7ff304033550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 --2- 172.21.15.169:0/697666387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0f52d0 0x7ff31c106640 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:54.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 >> 172.21.15.169:0/697666387 conn(0x7ff31c0f0a30 msgr2=0x7ff31c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:54.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 shutdown_connections 2024-01-11T05:08:54.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:54.023+0000 7ff321517700 1 -- 172.21.15.169:0/697666387 wait complete. 2024-01-11T05:08:54.024 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:54.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:54 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:54.021479+0000 mon.smithi067 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.169:0/697666387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:55.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:55.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:54.321451+0000 mgr.smithi067.srbzpc (mgr.14156) 174 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:56.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 -- 172.21.15.169:0/4024285943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 msgr2=0x7ff24c0fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:56.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 --2- 172.21.15.169:0/4024285943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c0fb5e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7ff240009fb0 tx=0x7ff240009b30).stop 2024-01-11T05:08:56.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 -- 172.21.15.169:0/4024285943 shutdown_connections 2024-01-11T05:08:56.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 --2- 172.21.15.169:0/4024285943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c0fb5e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 -- 172.21.15.169:0/4024285943 >> 172.21.15.169:0/4024285943 conn(0x7ff24c0f0be0 msgr2=0x7ff24c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 -- 172.21.15.169:0/4024285943 shutdown_connections 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.547+0000 7ff2531d6700 1 -- 172.21.15.169:0/4024285943 wait complete. 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.548+0000 7ff2531d6700 1 Processor -- start 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.548+0000 7ff2531d6700 1 -- start start 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.548+0000 7ff2531d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.548+0000 7ff2531d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c108e60 con 0x7ff24c0602e0 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff250f72700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff250f72700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60176/0 (socket says 172.21.15.169:60176) 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff250f72700 1 -- 172.21.15.169:0/2788812795 learned_addr learned my addr 172.21.15.169:0/2788812795 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:56.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff250f72700 1 -- 172.21.15.169:0/2788812795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff240005040 con 0x7ff24c0602e0 2024-01-11T05:08:56.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff250f72700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c108970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7ff240009f80 tx=0x7ff24000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:56.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff240009320 con 0x7ff24c0602e0 2024-01-11T05:08:56.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c109060 con 0x7ff24c0602e0 2024-01-11T05:08:56.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff240013680 con 0x7ff24c0602e0 2024-01-11T05:08:56.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff24001eda0 con 0x7ff24c0602e0 2024-01-11T05:08:56.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.549+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c1094f0 con 0x7ff24c0602e0 2024-01-11T05:08:56.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.550+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff24001e420 con 0x7ff24c0602e0 2024-01-11T05:08:56.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.550+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c045bb0 con 0x7ff24c0602e0 2024-01-11T05:08:56.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.550+0000 7ff2497fa700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff23c031090 0x7ff23c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:56.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.550+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff240042420 con 0x7ff24c0602e0 2024-01-11T05:08:56.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.550+0000 7ff24bfff700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff23c031090 0x7ff23c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:56.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.551+0000 7ff24bfff700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff23c031090 0x7ff23c033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff2380096a0 tx=0x7ff238006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:56.555 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.555+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff24001d380 con 0x7ff24c0602e0 2024-01-11T05:08:56.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.764+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff24c10a340 con 0x7ff24c0602e0 2024-01-11T05:08:56.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.765+0000 7ff2497fa700 1 -- 172.21.15.169:0/2788812795 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff24001cb10 con 0x7ff24c0602e0 2024-01-11T05:08:56.765 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:56.765 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:56.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff23c031090 msgr2=0x7ff23c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:56.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff23c031090 0x7ff23c033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff2380096a0 tx=0x7ff238006b40).stop 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 msgr2=0x7ff24c108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c108970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7ff240009f80 tx=0x7ff24000b040).stop 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 shutdown_connections 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff23c031090 0x7ff23c033550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 --2- 172.21.15.169:0/2788812795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c0602e0 0x7ff24c108970 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 >> 172.21.15.169:0/2788812795 conn(0x7ff24c0f0be0 msgr2=0x7ff24c0f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 shutdown_connections 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:56.766+0000 7ff2531d6700 1 -- 172.21.15.169:0/2788812795 wait complete. 2024-01-11T05:08:56.767 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:56.322081+0000 mgr.smithi067.srbzpc (mgr.14156) 175 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:08:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:56.765047+0000 mon.smithi067 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.169:0/2788812795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:08:58.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:08:58.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:08:59.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 -- 172.21.15.169:0/1950791666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 msgr2=0x7fba440f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:59.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 --2- 172.21.15.169:0/1950791666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba440f69b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fba38009fb0 tx=0x7fba38009b30).stop 2024-01-11T05:08:59.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 -- 172.21.15.169:0/1950791666 shutdown_connections 2024-01-11T05:08:59.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 --2- 172.21.15.169:0/1950791666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba440f69b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:59.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 -- 172.21.15.169:0/1950791666 >> 172.21.15.169:0/1950791666 conn(0x7fba440f1cf0 msgr2=0x7fba440f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:59.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 -- 172.21.15.169:0/1950791666 shutdown_connections 2024-01-11T05:08:59.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.267+0000 7fba49769700 1 -- 172.21.15.169:0/1950791666 wait complete. 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba49769700 1 Processor -- start 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba49769700 1 -- start start 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba49769700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba441067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba49769700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba44106cc0 con 0x7fba440f6590 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba42ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba441067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba42ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba441067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:60192/0 (socket says 172.21.15.169:60192) 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba42ffd700 1 -- 172.21.15.169:0/3867754452 learned_addr learned my addr 172.21.15.169:0/3867754452 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba42ffd700 1 -- 172.21.15.169:0/3867754452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38005040 con 0x7fba440f6590 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.268+0000 7fba42ffd700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba441067d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fba38000c00 tx=0x7fba3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.269+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba380095f0 con 0x7fba440f6590 2024-01-11T05:08:59.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.269+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba44106ec0 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.269+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fba38007e50 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.269+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba38014e70 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.269+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44107350 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.269+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fba38025070 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.270+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44045bb0 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.270+0000 7fba2b7fe700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba2c031090 0x7fba2c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.270+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fba38041e30 con 0x7fba440f6590 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.270+0000 7fba427fc700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba2c031090 0x7fba2c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:08:59.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.270+0000 7fba427fc700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba2c031090 0x7fba2c033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fba340096a0 tx=0x7fba34006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:08:59.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.274+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fba3800f4e0 con 0x7fba440f6590 2024-01-11T05:08:59.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.486+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba441081c0 con 0x7fba440f6590 2024-01-11T05:08:59.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.487+0000 7fba2b7fe700 1 -- 172.21.15.169:0/3867754452 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba3800f300 con 0x7fba440f6590 2024-01-11T05:08:59.488 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:08:59.488 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.488+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba2c031090 msgr2=0x7fba2c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.488+0000 7fba49769700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba2c031090 0x7fba2c033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fba340096a0 tx=0x7fba34006b40).stop 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.488+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 msgr2=0x7fba441067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.488+0000 7fba49769700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba441067d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fba38000c00 tx=0x7fba3800b040).stop 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.488+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 shutdown_connections 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.489+0000 7fba49769700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba2c031090 0x7fba2c033550 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.489+0000 7fba49769700 1 --2- 172.21.15.169:0/3867754452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba440f6590 0x7fba441067d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:08:59.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.489+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 >> 172.21.15.169:0/3867754452 conn(0x7fba440f1cf0 msgr2=0x7fba440f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:08:59.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.489+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 shutdown_connections 2024-01-11T05:08:59.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:08:59.489+0000 7fba49769700 1 -- 172.21.15.169:0/3867754452 wait complete. 2024-01-11T05:08:59.491 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:08:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:08:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:08:58.322479+0000 mgr.smithi067.srbzpc (mgr.14156) 176 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:00.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:00 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:08:59.487677+0000 mon.smithi067 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.169:0/3867754452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:00.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:00.897 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:00.322777+0000 mgr.smithi067.srbzpc (mgr.14156) 177 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:02.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.071+0000 7f13c4ba3700 1 -- 172.21.15.169:0/2821717038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 msgr2=0x7f13c00f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:02.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.071+0000 7f13c4ba3700 1 --2- 172.21.15.169:0/2821717038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c00f6800 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f13b0009fb0 tx=0x7f13b0009b30).stop 2024-01-11T05:09:02.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 -- 172.21.15.169:0/2821717038 shutdown_connections 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 --2- 172.21.15.169:0/2821717038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c00f6800 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 -- 172.21.15.169:0/2821717038 >> 172.21.15.169:0/2821717038 conn(0x7f13c00f1b40 msgr2=0x7f13c00f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 -- 172.21.15.169:0/2821717038 shutdown_connections 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 -- 172.21.15.169:0/2821717038 wait complete. 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 Processor -- start 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 -- start start 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c0106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:02.074 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.072+0000 7f13c4ba3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c0106b10 con 0x7f13c00f63e0 2024-01-11T05:09:02.075 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13bf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c0106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13bf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c0106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48150/0 (socket says 172.21.15.169:48150) 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13bf7fe700 1 -- 172.21.15.169:0/279160368 learned_addr learned my addr 172.21.15.169:0/279160368 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13bf7fe700 1 -- 172.21.15.169:0/279160368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0005040 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13bf7fe700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c0106620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f13b00089f0 tx=0x7f13b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13b0009320 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c0106d10 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f13b0013ec0 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13b0014d30 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.073+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c01071a0 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.074+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f13b00095f0 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.074+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c0045bb0 con 0x7f13c00f63e0 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.074+0000 7f13a7fff700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13a80310e0 0x7f13a80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:02.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.074+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13b00424a0 con 0x7f13c00f63e0 2024-01-11T05:09:02.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.075+0000 7f13beffd700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13a80310e0 0x7f13a80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:02.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.075+0000 7f13beffd700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13a80310e0 0x7f13a80335a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f13b40096a0 tx=0x7f13b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:02.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.079+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f13b001d7f0 con 0x7f13c00f63e0 2024-01-11T05:09:02.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.290+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13c0108050 con 0x7f13c00f63e0 2024-01-11T05:09:02.291 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.290+0000 7f13a7fff700 1 -- 172.21.15.169:0/279160368 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13b001d7f0 con 0x7f13c00f63e0 2024-01-11T05:09:02.292 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:02.292 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:02.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13a80310e0 msgr2=0x7f13a80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:02.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13a80310e0 0x7f13a80335a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f13b40096a0 tx=0x7f13b4006b40).stop 2024-01-11T05:09:02.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 msgr2=0x7f13c0106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:02.293 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c0106620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f13b00089f0 tx=0x7f13b000b040).stop 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 shutdown_connections 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f13a80310e0 0x7f13a80335a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 --2- 172.21.15.169:0/279160368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00f63e0 0x7f13c0106620 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 >> 172.21.15.169:0/279160368 conn(0x7f13c00f1b40 msgr2=0x7f13c00f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 shutdown_connections 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:02.292+0000 7f13c4ba3700 1 -- 172.21.15.169:0/279160368 wait complete. 2024-01-11T05:09:02.294 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:02.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:02.291770+0000 mon.smithi067 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.169:0/279160368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:03.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:03.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:02.323092+0000 mgr.smithi067.srbzpc (mgr.14156) 178 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 -- 172.21.15.169:0/3222377987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 msgr2=0x7fd5900f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 --2- 172.21.15.169:0/3222377987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f9190 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fd584009fb0 tx=0x7fd584009b30).stop 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 -- 172.21.15.169:0/3222377987 shutdown_connections 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 --2- 172.21.15.169:0/3222377987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f9190 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 -- 172.21.15.169:0/3222377987 >> 172.21.15.169:0/3222377987 conn(0x7fd5900f0c10 msgr2=0x7fd5900f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 -- 172.21.15.169:0/3222377987 shutdown_connections 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 -- 172.21.15.169:0/3222377987 wait complete. 2024-01-11T05:09:04.739 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.736+0000 7fd59621e700 1 Processor -- start 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd59621e700 1 -- start start 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd59621e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd59621e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900f5090 con 0x7fd590060360 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd58f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd58f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48178/0 (socket says 172.21.15.169:48178) 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd58f7fe700 1 -- 172.21.15.169:0/1267245023 learned_addr learned my addr 172.21.15.169:0/1267245023 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.737+0000 7fd58f7fe700 1 -- 172.21.15.169:0/1267245023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584005040 con 0x7fd590060360 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.738+0000 7fd58f7fe700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f6c10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fd584009f80 tx=0x7fd58400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.738+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd584009320 con 0x7fd590060360 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.738+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5900f5230 con 0x7fd590060360 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.738+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd584013680 con 0x7fd590060360 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.738+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58401eda0 con 0x7fd590060360 2024-01-11T05:09:04.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.738+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5900f56c0 con 0x7fd590060360 2024-01-11T05:09:04.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.739+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd58401e420 con 0x7fd590060360 2024-01-11T05:09:04.741 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.739+0000 7fd577fff700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd578031090 0x7fd578033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:04.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.739+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd584042350 con 0x7fd590060360 2024-01-11T05:09:04.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.739+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5900591e0 con 0x7fd590060360 2024-01-11T05:09:04.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.739+0000 7fd58effd700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd578031090 0x7fd578033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:04.742 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.739+0000 7fd58effd700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd578031090 0x7fd578033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd5800096a0 tx=0x7fd580006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:04.744 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.744+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd58401e700 con 0x7fd590060360 2024-01-11T05:09:04.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.949+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5900fc090 con 0x7fd590060360 2024-01-11T05:09:04.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.950+0000 7fd577fff700 1 -- 172.21.15.169:0/1267245023 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd58401cb10 con 0x7fd590060360 2024-01-11T05:09:04.951 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:04.951 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd578031090 msgr2=0x7fd578033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd578031090 0x7fd578033550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd5800096a0 tx=0x7fd580006b40).stop 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 msgr2=0x7fd5900f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f6c10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fd584009f80 tx=0x7fd58400b040).stop 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 shutdown_connections 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd578031090 0x7fd578033550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 --2- 172.21.15.169:0/1267245023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd590060360 0x7fd5900f6c10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.951+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 >> 172.21.15.169:0/1267245023 conn(0x7fd5900f0c10 msgr2=0x7fd5900f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.952+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 shutdown_connections 2024-01-11T05:09:04.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:04.952+0000 7fd59621e700 1 -- 172.21.15.169:0/1267245023 wait complete. 2024-01-11T05:09:04.954 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:04.323345+0000 mgr.smithi067.srbzpc (mgr.14156) 179 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:05.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:04.951100+0000 mon.smithi067 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.169:0/1267245023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:06.319 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:06.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:07.448 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 -- 172.21.15.169:0/1977575699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 msgr2=0x7f16040f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 --2- 172.21.15.169:0/1977575699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16040f69b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f15f4004760 tx=0x7f15f4009ad0).stop 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 -- 172.21.15.169:0/1977575699 shutdown_connections 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 --2- 172.21.15.169:0/1977575699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16040f69b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 -- 172.21.15.169:0/1977575699 >> 172.21.15.169:0/1977575699 conn(0x7f16040f1cd0 msgr2=0x7f16040f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 -- 172.21.15.169:0/1977575699 shutdown_connections 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 -- 172.21.15.169:0/1977575699 wait complete. 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 Processor -- start 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.447+0000 7f160a59a700 1 -- start start 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f160a59a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16041067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:07.449 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f160a59a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604106cb0 con 0x7f16040f6590 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1603fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16041067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1603fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16041067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:48204/0 (socket says 172.21.15.169:48204) 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1603fff700 1 -- 172.21.15.169:0/875132120 learned_addr learned my addr 172.21.15.169:0/875132120 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1603fff700 1 -- 172.21.15.169:0/875132120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f4009d70 con 0x7f16040f6590 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1603fff700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16041067c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f15f4004170 tx=0x7f15f4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15f400e040 con 0x7f16040f6590 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.448+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f15f4008ef0 con 0x7f16040f6590 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.449+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1604106eb0 con 0x7f16040f6590 2024-01-11T05:09:07.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.449+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15f4014c90 con 0x7f16040f6590 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.449+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1604107340 con 0x7f16040f6590 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.449+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f15f4014420 con 0x7f16040f6590 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.449+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1604045bb0 con 0x7f16040f6590 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.449+0000 7f1600ff9700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f15ec031090 0x7f15ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.450+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f15f4042240 con 0x7f16040f6590 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.450+0000 7f16037fe700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f15ec031090 0x7f15ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:07.452 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.450+0000 7f16037fe700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f15ec031090 0x7f15ec033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f15f80096a0 tx=0x7f15f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:07.455 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.455+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f15f4026300 con 0x7f16040f6590 2024-01-11T05:09:07.665 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.664+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16041081f0 con 0x7f16040f6590 2024-01-11T05:09:07.666 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.665+0000 7f1600ff9700 1 -- 172.21.15.169:0/875132120 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f15f4012b20 con 0x7f16040f6590 2024-01-11T05:09:07.666 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:07.666 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f15ec031090 msgr2=0x7f15ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f15ec031090 0x7f15ec033550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f15f80096a0 tx=0x7f15f8006b40).stop 2024-01-11T05:09:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 msgr2=0x7f16041067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16041067c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f15f4004170 tx=0x7f15f4009ad0).stop 2024-01-11T05:09:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 shutdown_connections 2024-01-11T05:09:07.668 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f15ec031090 0x7f15ec033550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:07.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 --2- 172.21.15.169:0/875132120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16040f6590 0x7f16041067c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:07.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 >> 172.21.15.169:0/875132120 conn(0x7f16040f1cd0 msgr2=0x7f16040f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:07.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 shutdown_connections 2024-01-11T05:09:07.669 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:07.667+0000 7f160a59a700 1 -- 172.21.15.169:0/875132120 wait complete. 2024-01-11T05:09:07.669 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:07.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:06.323679+0000 mgr.smithi067.srbzpc (mgr.14156) 180 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:08.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:08 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:07.666176+0000 mon.smithi067 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.169:0/875132120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:09.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:09.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:08.323925+0000 mgr.smithi067.srbzpc (mgr.14156) 181 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 -- 172.21.15.169:0/2930857084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 msgr2=0x7f5ee80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 --2- 172.21.15.169:0/2930857084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee80f58a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f5ed0009fb0 tx=0x7f5ed0009b30).stop 2024-01-11T05:09:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 -- 172.21.15.169:0/2930857084 shutdown_connections 2024-01-11T05:09:10.143 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 --2- 172.21.15.169:0/2930857084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee80f58a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 -- 172.21.15.169:0/2930857084 >> 172.21.15.169:0/2930857084 conn(0x7f5ee80f0be0 msgr2=0x7f5ee80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 -- 172.21.15.169:0/2930857084 shutdown_connections 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.141+0000 7f5eee61a700 1 -- 172.21.15.169:0/2930857084 wait complete. 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5eee61a700 1 Processor -- start 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5eee61a700 1 -- start start 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5eee61a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5eee61a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee8060300 con 0x7f5ee80f5480 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5ee7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5ee7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40502/0 (socket says 172.21.15.169:40502) 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5ee7fff700 1 -- 172.21.15.169:0/1995733287 learned_addr learned my addr 172.21.15.169:0/1995733287 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5ee7fff700 1 -- 172.21.15.169:0/1995733287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0005040 con 0x7f5ee80f5480 2024-01-11T05:09:10.144 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.142+0000 7f5ee7fff700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee8109510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f5ed00089f0 tx=0x7f5ed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:10.145 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.143+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ed0009320 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.143+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5ed0013ec0 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.143+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee8060500 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.143+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ed0014d30 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.143+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee8109bc0 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.143+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5ed00095f0 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.144+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee80591e0 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.144+0000 7f5ee4ff9700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ed4031090 0x7f5ed4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.144+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5ed00423e0 con 0x7f5ee80f5480 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.144+0000 7f5ee77fe700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ed4031090 0x7f5ed4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:10.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.144+0000 7f5ee77fe700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ed4031090 0x7f5ed4033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5ed80096a0 tx=0x7f5ed8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:10.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.149+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5ed001d980 con 0x7f5ee80f5480 2024-01-11T05:09:10.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.359+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ee8045bb0 con 0x7f5ee80f5480 2024-01-11T05:09:10.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.360+0000 7f5ee4ff9700 1 -- 172.21.15.169:0/1995733287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ed001d380 con 0x7f5ee80f5480 2024-01-11T05:09:10.361 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:10.361 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:10.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ed4031090 msgr2=0x7f5ed4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ed4031090 0x7f5ed4033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5ed80096a0 tx=0x7f5ed8006b40).stop 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 msgr2=0x7f5ee8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee8109510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f5ed00089f0 tx=0x7f5ed000b040).stop 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 shutdown_connections 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5ed4031090 0x7f5ed4033550 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 --2- 172.21.15.169:0/1995733287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80f5480 0x7f5ee8109510 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 >> 172.21.15.169:0/1995733287 conn(0x7f5ee80f0be0 msgr2=0x7f5ee80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 shutdown_connections 2024-01-11T05:09:10.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:10.361+0000 7f5eee61a700 1 -- 172.21.15.169:0/1995733287 wait complete. 2024-01-11T05:09:10.364 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:10.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:10 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:10.360811+0000 mon.smithi067 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.169:0/1995733287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:11.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:11.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:10.324205+0000 mgr.smithi067.srbzpc (mgr.14156) 182 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:12.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 -- 172.21.15.169:0/3031497211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 msgr2=0x7f54200f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 --2- 172.21.15.169:0/3031497211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200f58a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f5410009fb0 tx=0x7f5410009b30).stop 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 -- 172.21.15.169:0/3031497211 shutdown_connections 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 --2- 172.21.15.169:0/3031497211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200f58a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 -- 172.21.15.169:0/3031497211 >> 172.21.15.169:0/3031497211 conn(0x7f54200f0be0 msgr2=0x7f54200f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 -- 172.21.15.169:0/3031497211 shutdown_connections 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.861+0000 7f5426911700 1 -- 172.21.15.169:0/3031497211 wait complete. 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.862+0000 7f5426911700 1 Processor -- start 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.862+0000 7f5426911700 1 -- start start 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.862+0000 7f5426911700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.862+0000 7f5426911700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200fb8e0 con 0x7f54200f5480 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:12.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40526/0 (socket says 172.21.15.169:40526) 2024-01-11T05:09:12.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541ffff700 1 -- 172.21.15.169:0/1110076809 learned_addr learned my addr 172.21.15.169:0/1110076809 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541ffff700 1 -- 172.21.15.169:0/1110076809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5410005040 con 0x7f54200f5480 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541ffff700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200fd460 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f54100089f0 tx=0x7f541000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5410009320 con 0x7f54200f5480 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54200fba80 con 0x7f54200f5480 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5410013ec0 con 0x7f54200f5480 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5410014d30 con 0x7f54200f5480 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.863+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54200fbf10 con 0x7f54200f5480 2024-01-11T05:09:12.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.864+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f54100095f0 con 0x7f54200f5480 2024-01-11T05:09:12.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.864+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54200591e0 con 0x7f54200f5480 2024-01-11T05:09:12.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.865+0000 7f541cff9700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f54080310e0 0x7f54080335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:12.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.865+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f54100424e0 con 0x7f54200f5480 2024-01-11T05:09:12.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.865+0000 7f541f7fe700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f54080310e0 0x7f54080335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:12.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.865+0000 7f541f7fe700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f54080310e0 0x7f54080335a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f54140096a0 tx=0x7f5414006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:12.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:12.870+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f54100268c0 con 0x7f54200f5480 2024-01-11T05:09:13.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.079+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5420045bb0 con 0x7f54200f5480 2024-01-11T05:09:13.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.080+0000 7f541cff9700 1 -- 172.21.15.169:0/1110076809 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f541000fe40 con 0x7f54200f5480 2024-01-11T05:09:13.081 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:13.081 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f54080310e0 msgr2=0x7f54080335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f54080310e0 0x7f54080335a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f54140096a0 tx=0x7f5414006b40).stop 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 msgr2=0x7f54200fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200fd460 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f54100089f0 tx=0x7f541000b040).stop 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 shutdown_connections 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f54080310e0 0x7f54080335a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 --2- 172.21.15.169:0/1110076809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54200f5480 0x7f54200fd460 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 >> 172.21.15.169:0/1110076809 conn(0x7f54200f0be0 msgr2=0x7f54200f17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 shutdown_connections 2024-01-11T05:09:13.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:13.082+0000 7f5426911700 1 -- 172.21.15.169:0/1110076809 wait complete. 2024-01-11T05:09:13.084 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:12.324528+0000 mgr.smithi067.srbzpc (mgr.14156) 183 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:13.081378+0000 mon.smithi067 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.169:0/1110076809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:14.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:14.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:15.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:14.324824+0000 mgr.smithi067.srbzpc (mgr.14156) 184 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:15.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 -- 172.21.15.169:0/1107229996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 msgr2=0x7ff3780f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:15.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 --2- 172.21.15.169:0/1107229996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff3780f9160 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff368009fb0 tx=0x7ff368009b30).stop 2024-01-11T05:09:15.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 -- 172.21.15.169:0/1107229996 shutdown_connections 2024-01-11T05:09:15.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 --2- 172.21.15.169:0/1107229996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff3780f9160 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:15.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 -- 172.21.15.169:0/1107229996 >> 172.21.15.169:0/1107229996 conn(0x7ff3780f0be0 msgr2=0x7ff3780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:15.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 -- 172.21.15.169:0/1107229996 shutdown_connections 2024-01-11T05:09:15.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.539+0000 7ff37fd0a700 1 -- 172.21.15.169:0/1107229996 wait complete. 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37fd0a700 1 Processor -- start 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37fd0a700 1 -- start start 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37fd0a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff378106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37fd0a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378106c40 con 0x7ff378060360 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37daa6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff378106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37daa6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff378106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40554/0 (socket says 172.21.15.169:40554) 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37daa6700 1 -- 172.21.15.169:0/3301767970 learned_addr learned my addr 172.21.15.169:0/3301767970 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.540+0000 7ff37daa6700 1 -- 172.21.15.169:0/3301767970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368005040 con 0x7ff378060360 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.541+0000 7ff37daa6700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff378106750 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff368009f80 tx=0x7ff36800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:15.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.541+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff368009320 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.541+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff378106e40 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.541+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff368013680 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.541+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff36801eda0 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.541+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3781072d0 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.542+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff36801e420 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.542+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff378045bb0 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.542+0000 7ff36e7fc700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff364031090 0x7ff364033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.542+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff368042440 con 0x7ff378060360 2024-01-11T05:09:15.544 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.542+0000 7ff37d2a5700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff364031090 0x7ff364033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:15.545 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.542+0000 7ff37d2a5700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff364031090 0x7ff364033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ff3740096a0 tx=0x7ff374006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:15.548 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.547+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff36801d380 con 0x7ff378060360 2024-01-11T05:09:15.754 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.753+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff378108180 con 0x7ff378060360 2024-01-11T05:09:15.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.754+0000 7ff36e7fc700 1 -- 172.21.15.169:0/3301767970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff368004490 con 0x7ff378060360 2024-01-11T05:09:15.755 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:15.755 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff364031090 msgr2=0x7ff364033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff364031090 0x7ff364033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ff3740096a0 tx=0x7ff374006b40).stop 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 msgr2=0x7ff378106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff378106750 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff368009f80 tx=0x7ff36800b040).stop 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 shutdown_connections 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff364031090 0x7ff364033550 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 --2- 172.21.15.169:0/3301767970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff378060360 0x7ff378106750 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 >> 172.21.15.169:0/3301767970 conn(0x7ff3780f0be0 msgr2=0x7ff3780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:15.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 shutdown_connections 2024-01-11T05:09:15.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:15.756+0000 7ff37fd0a700 1 -- 172.21.15.169:0/3301767970 wait complete. 2024-01-11T05:09:15.758 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:16.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:16 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:15.755493+0000 mon.smithi067 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.169:0/3301767970' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:17.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:17.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:17.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:16.325051+0000 mgr.smithi067.srbzpc (mgr.14156) 185 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 -- 172.21.15.169:0/3186699624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 msgr2=0x7f1d2c0f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 --2- 172.21.15.169:0/3186699624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c0f5e80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f1d28009fb0 tx=0x7f1d28009b30).stop 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 -- 172.21.15.169:0/3186699624 shutdown_connections 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 --2- 172.21.15.169:0/3186699624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c0f5e80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 -- 172.21.15.169:0/3186699624 >> 172.21.15.169:0/3186699624 conn(0x7f1d2c0f0390 msgr2=0x7f1d2c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 -- 172.21.15.169:0/3186699624 shutdown_connections 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 -- 172.21.15.169:0/3186699624 wait complete. 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.278+0000 7f1d33c62700 1 Processor -- start 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d33c62700 1 -- start start 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d33c62700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:18.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d33c62700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c106cd0 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d319fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d319fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:40582/0 (socket says 172.21.15.169:40582) 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d319fe700 1 -- 172.21.15.169:0/1187207837 learned_addr learned my addr 172.21.15.169:0/1187207837 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d319fe700 1 -- 172.21.15.169:0/1187207837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d28005040 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d319fe700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c1067e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f1d280089f0 tx=0x7f1d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d28009320 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1d28013ec0 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c106ed0 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.279+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d28014d30 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.282 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.280+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c107360 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.280+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1d28014420 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.280+0000 7f1d227fc700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1d18031090 0x7f1d18033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.280+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1d28042400 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.280+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c045bb0 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.281+0000 7f1d311fd700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1d18031090 0x7f1d18033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:18.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.281+0000 7f1d311fd700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1d18031090 0x7f1d18033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f1d1c0096a0 tx=0x7f1d1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:18.286 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.285+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1d28014750 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.498+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d2c108160 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.499+0000 7f1d227fc700 1 -- 172.21.15.169:0/1187207837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d2801d380 con 0x7f1d2c0f5a60 2024-01-11T05:09:18.500 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:18.501 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1d18031090 msgr2=0x7f1d18033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1d18031090 0x7f1d18033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f1d1c0096a0 tx=0x7f1d1c006b40).stop 2024-01-11T05:09:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 msgr2=0x7f1d2c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c1067e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f1d280089f0 tx=0x7f1d2800b040).stop 2024-01-11T05:09:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 shutdown_connections 2024-01-11T05:09:18.502 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1d18031090 0x7f1d18033550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:18.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 --2- 172.21.15.169:0/1187207837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d2c0f5a60 0x7f1d2c1067e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:18.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 >> 172.21.15.169:0/1187207837 conn(0x7f1d2c0f0390 msgr2=0x7f1d2c0f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:18.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 shutdown_connections 2024-01-11T05:09:18.503 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:18.501+0000 7f1d33c62700 1 -- 172.21.15.169:0/1187207837 wait complete. 2024-01-11T05:09:18.503 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:18.325373+0000 mgr.smithi067.srbzpc (mgr.14156) 186 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:18.500615+0000 mon.smithi067 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.169:0/1187207837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:19.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:19.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 -- 172.21.15.169:0/600242812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 msgr2=0x7f2fe00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 --2- 172.21.15.169:0/600242812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe00f58a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f2fc8009fb0 tx=0x7f2fc8009b30).stop 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 -- 172.21.15.169:0/600242812 shutdown_connections 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 --2- 172.21.15.169:0/600242812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe00f58a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 -- 172.21.15.169:0/600242812 >> 172.21.15.169:0/600242812 conn(0x7f2fe00f0be0 msgr2=0x7f2fe00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 -- 172.21.15.169:0/600242812 shutdown_connections 2024-01-11T05:09:20.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.950+0000 7f2fe4a39700 1 -- 172.21.15.169:0/600242812 wait complete. 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fe4a39700 1 Processor -- start 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fe4a39700 1 -- start start 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fe4a39700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fe4a39700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe0105be0 con 0x7f2fe00f5480 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fde59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fde59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54812/0 (socket says 172.21.15.169:54812) 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fde59c700 1 -- 172.21.15.169:0/4005727983 learned_addr learned my addr 172.21.15.169:0/4005727983 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fde59c700 1 -- 172.21.15.169:0/4005727983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8005040 con 0x7f2fe00f5480 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.951+0000 7f2fde59c700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe01056f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f2fc8000c00 tx=0x7f2fc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.952+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc80095f0 con 0x7f2fe00f5480 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.952+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe0105de0 con 0x7f2fe00f5480 2024-01-11T05:09:20.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.952+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2fc8007e50 con 0x7f2fe00f5480 2024-01-11T05:09:20.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.952+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fc8014e70 con 0x7f2fe00f5480 2024-01-11T05:09:20.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.952+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe0106270 con 0x7f2fe00f5480 2024-01-11T05:09:20.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.953+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2fc8025070 con 0x7f2fe00f5480 2024-01-11T05:09:20.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.953+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe0045bb0 con 0x7f2fe00f5480 2024-01-11T05:09:20.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.953+0000 7f2fd6ffd700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fcc031090 0x7f2fcc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:20.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.953+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2fc8041d40 con 0x7f2fe00f5480 2024-01-11T05:09:20.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.953+0000 7f2fddd9b700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fcc031090 0x7f2fcc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:20.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.953+0000 7f2fddd9b700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fcc031090 0x7f2fcc033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2fd00096a0 tx=0x7f2fd0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:20.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:20.958+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2fc800f4e0 con 0x7f2fe00f5480 2024-01-11T05:09:21.165 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.164+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fe00faf70 con 0x7f2fe00f5480 2024-01-11T05:09:21.166 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.165+0000 7f2fd6ffd700 1 -- 172.21.15.169:0/4005727983 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fc800f300 con 0x7f2fe00f5480 2024-01-11T05:09:21.166 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:21.166 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fcc031090 msgr2=0x7f2fcc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fcc031090 0x7f2fcc033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2fd00096a0 tx=0x7f2fd0006b40).stop 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 msgr2=0x7f2fe01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe01056f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f2fc8000c00 tx=0x7f2fc800b040).stop 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 shutdown_connections 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fcc031090 0x7f2fcc033550 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 --2- 172.21.15.169:0/4005727983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe00f5480 0x7f2fe01056f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 >> 172.21.15.169:0/4005727983 conn(0x7f2fe00f0be0 msgr2=0x7f2fe00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 shutdown_connections 2024-01-11T05:09:21.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:21.167+0000 7f2fe4a39700 1 -- 172.21.15.169:0/4005727983 wait complete. 2024-01-11T05:09:21.169 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:20.325706+0000 mgr.smithi067.srbzpc (mgr.14156) 187 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:21.166423+0000 mon.smithi067 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.169:0/4005727983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:22.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:22.533 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.655+0000 7f1833e48700 1 -- 172.21.15.169:0/732811793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 msgr2=0x7f182c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.655+0000 7f1833e48700 1 --2- 172.21.15.169:0/732811793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0f58a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f181c009fb0 tx=0x7f181c009b30).stop 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 -- 172.21.15.169:0/732811793 shutdown_connections 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 --2- 172.21.15.169:0/732811793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0f58a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 -- 172.21.15.169:0/732811793 >> 172.21.15.169:0/732811793 conn(0x7f182c0f0be0 msgr2=0x7f182c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 -- 172.21.15.169:0/732811793 shutdown_connections 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 -- 172.21.15.169:0/732811793 wait complete. 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 Processor -- start 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.656+0000 7f1833e48700 1 -- start start 2024-01-11T05:09:23.658 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1833e48700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0fb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:23.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1833e48700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c0f9a70 con 0x7f182c0f5480 2024-01-11T05:09:23.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1831be4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0fb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:23.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1831be4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0fb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54836/0 (socket says 172.21.15.169:54836) 2024-01-11T05:09:23.659 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1831be4700 1 -- 172.21.15.169:0/2582305448 learned_addr learned my addr 172.21.15.169:0/2582305448 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:23.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1831be4700 1 -- 172.21.15.169:0/2582305448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c005040 con 0x7f182c0f5480 2024-01-11T05:09:23.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1831be4700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0fb5f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f181c0089f0 tx=0x7f181c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:23.660 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f181c009320 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c0f9c10 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f181c013ec0 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.657+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f181c014d30 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.658+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c0fa0a0 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.658+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f181c0095f0 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.659+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c0591e0 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.659+0000 7f18227fc700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1818031090 0x7f1818033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.659+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f181c0423e0 con 0x7f182c0f5480 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.659+0000 7f18313e3700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1818031090 0x7f1818033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:23.661 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.659+0000 7f18313e3700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1818031090 0x7f1818033550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f18280096a0 tx=0x7f1828006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:23.664 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.664+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f181c01d7f0 con 0x7f182c0f5480 2024-01-11T05:09:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:22.325981+0000 mgr.smithi067.srbzpc (mgr.14156) 188 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:22.417083+0000 mon.smithi067 (mon.0) 252 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:09:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:22.417345+0000 mon.smithi067 (mon.0) 253 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:09:23.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.871+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f182c0fc0a0 con 0x7f182c0f5480 2024-01-11T05:09:23.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.871+0000 7f18227fc700 1 -- 172.21.15.169:0/2582305448 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f181c01d7f0 con 0x7f182c0f5480 2024-01-11T05:09:23.872 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:23.873 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1818031090 msgr2=0x7f1818033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1818031090 0x7f1818033550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f18280096a0 tx=0x7f1828006b40).stop 2024-01-11T05:09:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 msgr2=0x7f182c0fb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0fb5f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f181c0089f0 tx=0x7f181c00b040).stop 2024-01-11T05:09:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 shutdown_connections 2024-01-11T05:09:23.874 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1818031090 0x7f1818033550 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:23.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 --2- 172.21.15.169:0/2582305448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f182c0f5480 0x7f182c0fb5f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:23.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 >> 172.21.15.169:0/2582305448 conn(0x7f182c0f0be0 msgr2=0x7f182c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:23.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 shutdown_connections 2024-01-11T05:09:23.875 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:23.873+0000 7f1833e48700 1 -- 172.21.15.169:0/2582305448 wait complete. 2024-01-11T05:09:23.875 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:24.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:24 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:23.872646+0000 mon.smithi067 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.169:0/2582305448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:25.244 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:25.244 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:25.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:24.326311+0000 mgr.smithi067.srbzpc (mgr.14156) 189 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:26.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 -- 172.21.15.169:0/138924395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 msgr2=0x7f2ff40f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:26.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 --2- 172.21.15.169:0/138924395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff40f9050 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f2fe4009fb0 tx=0x7f2fe4009b30).stop 2024-01-11T05:09:26.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 -- 172.21.15.169:0/138924395 shutdown_connections 2024-01-11T05:09:26.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 --2- 172.21.15.169:0/138924395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff40f9050 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:26.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 -- 172.21.15.169:0/138924395 >> 172.21.15.169:0/138924395 conn(0x7f2ff40f0a30 msgr2=0x7f2ff40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:26.334 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 -- 172.21.15.169:0/138924395 shutdown_connections 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 -- 172.21.15.169:0/138924395 wait complete. 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.330+0000 7f2ff906b700 1 Processor -- start 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff906b700 1 -- start start 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff906b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff41065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff906b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4106aa0 con 0x7f2ff40602c0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff41065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff41065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54854/0 (socket says 172.21.15.169:54854) 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff3fff700 1 -- 172.21.15.169:0/1002227308 learned_addr learned my addr 172.21.15.169:0/1002227308 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff3fff700 1 -- 172.21.15.169:0/1002227308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe4005040 con 0x7f2ff40602c0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff3fff700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff41065b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f2fe4013fd0 tx=0x7f2fe4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe4018070 con 0x7f2ff40602c0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff4106ca0 con 0x7f2ff40602c0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2fe4007b10 con 0x7f2ff40602c0 2024-01-11T05:09:26.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.331+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe401c3c0 con 0x7f2ff40602c0 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.332+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff4107130 con 0x7f2ff40602c0 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.332+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2fe4007c80 con 0x7f2ff40602c0 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.332+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff4045bb0 con 0x7f2ff40602c0 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.332+0000 7f2ff0ff9700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fdc031090 0x7f2fdc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.333+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2fe4042a30 con 0x7f2ff40602c0 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.333+0000 7f2ff37fe700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fdc031090 0x7f2fdc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:26.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.333+0000 7f2ff37fe700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fdc031090 0x7f2fdc033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2fe80096a0 tx=0x7f2fe8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:26.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.337+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2fe4025b40 con 0x7f2ff40602c0 2024-01-11T05:09:26.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.548+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ff4107f80 con 0x7f2ff40602c0 2024-01-11T05:09:26.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.549+0000 7f2ff0ff9700 1 -- 172.21.15.169:0/1002227308 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fe4025b40 con 0x7f2ff40602c0 2024-01-11T05:09:26.550 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:26.550 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:26.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fdc031090 msgr2=0x7f2fdc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:26.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fdc031090 0x7f2fdc033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2fe80096a0 tx=0x7f2fe8006b40).stop 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 msgr2=0x7f2ff41065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff41065b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f2fe4013fd0 tx=0x7f2fe4009b30).stop 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 shutdown_connections 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2fdc031090 0x7f2fdc033550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 --2- 172.21.15.169:0/1002227308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ff40602c0 0x7f2ff41065b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 >> 172.21.15.169:0/1002227308 conn(0x7f2ff40f0a30 msgr2=0x7f2ff40f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 shutdown_connections 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:26.550+0000 7f2ff906b700 1 -- 172.21.15.169:0/1002227308 wait complete. 2024-01-11T05:09:26.552 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:26.326714+0000 mgr.smithi067.srbzpc (mgr.14156) 190 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:27.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:26.550116+0000 mon.smithi067 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.169:0/1002227308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:27.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:27.928 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:29.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- 172.21.15.169:0/1153061560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 msgr2=0x7fa1040f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:29.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 --2- 172.21.15.169:0/1153061560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa1040f6800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa0f4009fb0 tx=0x7fa0f4009b30).stop 2024-01-11T05:09:29.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- 172.21.15.169:0/1153061560 shutdown_connections 2024-01-11T05:09:29.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 --2- 172.21.15.169:0/1153061560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa1040f6800 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:29.042 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- 172.21.15.169:0/1153061560 >> 172.21.15.169:0/1153061560 conn(0x7fa1040f1b40 msgr2=0x7fa1040f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- 172.21.15.169:0/1153061560 shutdown_connections 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- 172.21.15.169:0/1153061560 wait complete. 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 Processor -- start 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- start start 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa104106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.040+0000 7fa10a836700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104106b50 con 0x7fa1040f63e0 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa109834700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa104106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa109834700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa104106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:54858/0 (socket says 172.21.15.169:54858) 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa109834700 1 -- 172.21.15.169:0/92979585 learned_addr learned my addr 172.21.15.169:0/92979585 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa109834700 1 -- 172.21.15.169:0/92979585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4005040 con 0x7fa1040f63e0 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa109834700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa104106660 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa0f40089f0 tx=0x7fa0f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f4009320 con 0x7fa1040f63e0 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa104106d50 con 0x7fa1040f63e0 2024-01-11T05:09:29.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa0f4013ec0 con 0x7fa1040f63e0 2024-01-11T05:09:29.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0f4014d30 con 0x7fa1040f63e0 2024-01-11T05:09:29.044 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.041+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1041071e0 con 0x7fa1040f63e0 2024-01-11T05:09:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.042+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa0f40263a0 con 0x7fa1040f63e0 2024-01-11T05:09:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.042+0000 7fa0fa7fc700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa0f0031090 0x7fa0f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.042+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa0f40423c0 con 0x7fa1040f63e0 2024-01-11T05:09:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.042+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104045bb0 con 0x7fa1040f63e0 2024-01-11T05:09:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.042+0000 7fa109033700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa0f0031090 0x7fa0f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:29.045 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.043+0000 7fa109033700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa0f0031090 0x7fa0f0033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa1000096a0 tx=0x7fa100006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:29.048 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.047+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa0f40266d0 con 0x7fa1040f63e0 2024-01-11T05:09:29.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.254+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa104107f20 con 0x7fa1040f63e0 2024-01-11T05:09:29.255 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.255+0000 7fa0fa7fc700 1 -- 172.21.15.169:0/92979585 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0f4012330 con 0x7fa1040f63e0 2024-01-11T05:09:29.256 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:29.256 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:29.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa0f0031090 msgr2=0x7fa0f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:29.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa0f0031090 0x7fa0f0033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa1000096a0 tx=0x7fa100006b40).stop 2024-01-11T05:09:29.257 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 msgr2=0x7fa104106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa104106660 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa0f40089f0 tx=0x7fa0f400b040).stop 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 shutdown_connections 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa0f0031090 0x7fa0f0033550 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 --2- 172.21.15.169:0/92979585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1040f63e0 0x7fa104106660 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 >> 172.21.15.169:0/92979585 conn(0x7fa1040f1b40 msgr2=0x7fa1040f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 shutdown_connections 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:29.256+0000 7fa10a836700 1 -- 172.21.15.169:0/92979585 wait complete. 2024-01-11T05:09:29.258 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:29.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:28.326976+0000 mgr.smithi067.srbzpc (mgr.14156) 191 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:29.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:29.256027+0000 mon.smithi067 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.169:0/92979585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:30.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:30.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:31.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 -- 172.21.15.169:0/1927187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 msgr2=0x7f1c2c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:31.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 --2- 172.21.15.169:0/1927187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c0f69b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f1c20009fb0 tx=0x7f1c20009b30).stop 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 -- 172.21.15.169:0/1927187431 shutdown_connections 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 --2- 172.21.15.169:0/1927187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c0f69b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 -- 172.21.15.169:0/1927187431 >> 172.21.15.169:0/1927187431 conn(0x7f1c2c0f1cf0 msgr2=0x7f1c2c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 -- 172.21.15.169:0/1927187431 shutdown_connections 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 -- 172.21.15.169:0/1927187431 wait complete. 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 Processor -- start 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.732+0000 7f1c32f9e700 1 -- start start 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c32f9e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c32f9e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c106c70 con 0x7f1c2c0f6590 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c30d3a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c30d3a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43628/0 (socket says 172.21.15.169:43628) 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c30d3a700 1 -- 172.21.15.169:0/3994222585 learned_addr learned my addr 172.21.15.169:0/3994222585 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c30d3a700 1 -- 172.21.15.169:0/3994222585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c20005040 con 0x7f1c2c0f6590 2024-01-11T05:09:31.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c30d3a700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c106780 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1c200089f0 tx=0x7f1c2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:31.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.733+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c20009320 con 0x7f1c2c0f6590 2024-01-11T05:09:31.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.734+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c106e70 con 0x7f1c2c0f6590 2024-01-11T05:09:31.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.734+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1c20013ec0 con 0x7f1c2c0f6590 2024-01-11T05:09:31.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.734+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c20014d30 con 0x7f1c2c0f6590 2024-01-11T05:09:31.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.734+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c107300 con 0x7f1c2c0f6590 2024-01-11T05:09:31.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.734+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1c200095f0 con 0x7f1c2c0f6590 2024-01-11T05:09:31.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.735+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c045bb0 con 0x7f1c2c0f6590 2024-01-11T05:09:31.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.735+0000 7f1c297fa700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1c1c031090 0x7f1c1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:31.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.735+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1c200424a0 con 0x7f1c2c0f6590 2024-01-11T05:09:31.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.735+0000 7f1c2bfff700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1c1c031090 0x7f1c1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:31.737 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.735+0000 7f1c2bfff700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1c1c031090 0x7f1c1c033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1c180096a0 tx=0x7f1c18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:31.740 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.740+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1c2001c320 con 0x7f1c2c0f6590 2024-01-11T05:09:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:30.327297+0000 mgr.smithi067.srbzpc (mgr.14156) 192 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:31.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.949+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c2c1081b0 con 0x7f1c2c0f6590 2024-01-11T05:09:31.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.950+0000 7f1c297fa700 1 -- 172.21.15.169:0/3994222585 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c2000f340 con 0x7f1c2c0f6590 2024-01-11T05:09:31.951 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:31.952 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1c1c031090 msgr2=0x7f1c1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1c1c031090 0x7f1c1c033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1c180096a0 tx=0x7f1c18006b40).stop 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 msgr2=0x7f1c2c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c106780 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1c200089f0 tx=0x7f1c2000b040).stop 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 shutdown_connections 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1c1c031090 0x7f1c1c033550 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 --2- 172.21.15.169:0/3994222585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c0f6590 0x7f1c2c106780 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 >> 172.21.15.169:0/3994222585 conn(0x7f1c2c0f1cf0 msgr2=0x7f1c2c0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:31.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 shutdown_connections 2024-01-11T05:09:31.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:31.952+0000 7f1c32f9e700 1 -- 172.21.15.169:0/3994222585 wait complete. 2024-01-11T05:09:31.954 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:32.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:32 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:31.951275+0000 mon.smithi067 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.169:0/3994222585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:33.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:33.313 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:32.327620+0000 mgr.smithi067.srbzpc (mgr.14156) 193 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.414+0000 7faca8398700 1 -- 172.21.15.169:0/2068031907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 msgr2=0x7faca00f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.414+0000 7faca8398700 1 --2- 172.21.15.169:0/2068031907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca00f4c60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fac9c009fb0 tx=0x7fac9c009b30).stop 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 -- 172.21.15.169:0/2068031907 shutdown_connections 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 --2- 172.21.15.169:0/2068031907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca00f4c60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 -- 172.21.15.169:0/2068031907 >> 172.21.15.169:0/2068031907 conn(0x7faca00f0360 msgr2=0x7faca00f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 -- 172.21.15.169:0/2068031907 shutdown_connections 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 -- 172.21.15.169:0/2068031907 wait complete. 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 Processor -- start 2024-01-11T05:09:34.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.415+0000 7faca8398700 1 -- start start 2024-01-11T05:09:34.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca8398700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:34.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca8398700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca0106cb0 con 0x7faca00f4840 2024-01-11T05:09:34.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca6134700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:34.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca6134700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43644/0 (socket says 172.21.15.169:43644) 2024-01-11T05:09:34.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca6134700 1 -- 172.21.15.169:0/3908139959 learned_addr learned my addr 172.21.15.169:0/3908139959 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca6134700 1 -- 172.21.15.169:0/3908139959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c005040 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7faca6134700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca01067c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fac9c0089f0 tx=0x7fac9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.416+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac9c009320 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.417+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca0106eb0 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.417+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fac9c013ec0 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.417+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac9c014d30 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.417+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca0107340 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.417+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fac9c0095f0 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.417+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca0045bb0 con 0x7faca00f4840 2024-01-11T05:09:34.420 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.418+0000 7fac92ffd700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fac8c031090 0x7fac8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:34.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.418+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fac9c0424a0 con 0x7faca00f4840 2024-01-11T05:09:34.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.418+0000 7faca5933700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fac8c031090 0x7fac8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:34.421 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.418+0000 7faca5933700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fac8c031090 0x7fac8c033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fac940096a0 tx=0x7fac94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:34.423 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.422+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fac9c01d980 con 0x7faca00f4840 2024-01-11T05:09:34.634 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.633+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faca01081f0 con 0x7faca00f4840 2024-01-11T05:09:34.635 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.635+0000 7fac92ffd700 1 -- 172.21.15.169:0/3908139959 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac9c01d380 con 0x7faca00f4840 2024-01-11T05:09:34.635 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:34.635 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fac8c031090 msgr2=0x7fac8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fac8c031090 0x7fac8c033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fac940096a0 tx=0x7fac94006b40).stop 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 msgr2=0x7faca01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca01067c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fac9c0089f0 tx=0x7fac9c00b040).stop 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 shutdown_connections 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fac8c031090 0x7fac8c033550 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 --2- 172.21.15.169:0/3908139959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00f4840 0x7faca01067c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 >> 172.21.15.169:0/3908139959 conn(0x7faca00f0360 msgr2=0x7faca00f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 shutdown_connections 2024-01-11T05:09:34.637 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:34.636+0000 7faca8398700 1 -- 172.21.15.169:0/3908139959 wait complete. 2024-01-11T05:09:34.638 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:34.328001+0000 mgr.smithi067.srbzpc (mgr.14156) 194 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:34.635073+0000 mon.smithi067 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.169:0/3908139959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:35.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:35.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:37.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 -- 172.21.15.169:0/3136228952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 msgr2=0x7ff2340f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:37.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 --2- 172.21.15.169:0/3136228952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2340f69b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ff228004760 tx=0x7ff228009ad0).stop 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 -- 172.21.15.169:0/3136228952 shutdown_connections 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 --2- 172.21.15.169:0/3136228952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2340f69b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 -- 172.21.15.169:0/3136228952 >> 172.21.15.169:0/3136228952 conn(0x7ff2340f1cd0 msgr2=0x7ff2340f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 -- 172.21.15.169:0/3136228952 shutdown_connections 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 -- 172.21.15.169:0/3136228952 wait complete. 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.105+0000 7ff239cca700 1 Processor -- start 2024-01-11T05:09:37.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff239cca700 1 -- start start 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff239cca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2341067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff239cca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff234106cb0 con 0x7ff2340f6590 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2341067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2341067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43662/0 (socket says 172.21.15.169:43662) 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff2337fe700 1 -- 172.21.15.169:0/375545258 learned_addr learned my addr 172.21.15.169:0/375545258 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.106+0000 7ff2337fe700 1 -- 172.21.15.169:0/375545258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228009d70 con 0x7ff2340f6590 2024-01-11T05:09:37.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.107+0000 7ff2337fe700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2341067c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff228004170 tx=0x7ff228009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:37.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.107+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff22800e040 con 0x7ff2340f6590 2024-01-11T05:09:37.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.107+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff234106eb0 con 0x7ff2340f6590 2024-01-11T05:09:37.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.107+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff228008ef0 con 0x7ff2340f6590 2024-01-11T05:09:37.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.107+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff228014c90 con 0x7ff2340f6590 2024-01-11T05:09:37.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.107+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff234107340 con 0x7ff2340f6590 2024-01-11T05:09:37.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.108+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff228014420 con 0x7ff2340f6590 2024-01-11T05:09:37.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.108+0000 7ff21bfff700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff21c031090 0x7ff21c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:37.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.108+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff2280422b0 con 0x7ff2340f6590 2024-01-11T05:09:37.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.108+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff234045bb0 con 0x7ff2340f6590 2024-01-11T05:09:37.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.108+0000 7ff232ffd700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff21c031090 0x7ff21c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:37.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.108+0000 7ff232ffd700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff21c031090 0x7ff21c033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff2240096a0 tx=0x7ff224006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:37.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.113+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff228026300 con 0x7ff2340f6590 2024-01-11T05:09:37.327 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.326+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff234108140 con 0x7ff2340f6590 2024-01-11T05:09:37.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.327+0000 7ff21bfff700 1 -- 172.21.15.169:0/375545258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff228012b20 con 0x7ff2340f6590 2024-01-11T05:09:37.328 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:37.329 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:37.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff21c031090 msgr2=0x7ff21c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:37.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff21c031090 0x7ff21c033550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff2240096a0 tx=0x7ff224006b40).stop 2024-01-11T05:09:37.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 msgr2=0x7ff2341067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:37.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2341067c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff228004170 tx=0x7ff228009ad0).stop 2024-01-11T05:09:37.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 shutdown_connections 2024-01-11T05:09:37.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff21c031090 0x7ff21c033550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:37.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 --2- 172.21.15.169:0/375545258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2340f6590 0x7ff2341067c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:37.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 >> 172.21.15.169:0/375545258 conn(0x7ff2340f1cd0 msgr2=0x7ff2340f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:37.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 shutdown_connections 2024-01-11T05:09:37.331 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:37.329+0000 7ff239cca700 1 -- 172.21.15.169:0/375545258 wait complete. 2024-01-11T05:09:37.331 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:36.328377+0000 mgr.smithi067.srbzpc (mgr.14156) 195 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:37.328453+0000 mon.smithi067 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.169:0/375545258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:38.699 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:38.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:38.328785+0000 mgr.smithi067.srbzpc (mgr.14156) 196 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 -- 172.21.15.169:0/1696006111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 msgr2=0x7fb40c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 --2- 172.21.15.169:0/1696006111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c0f58a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fb3f4009fb0 tx=0x7fb3f4009b30).stop 2024-01-11T05:09:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 -- 172.21.15.169:0/1696006111 shutdown_connections 2024-01-11T05:09:39.815 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 --2- 172.21.15.169:0/1696006111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c0f58a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:39.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 -- 172.21.15.169:0/1696006111 >> 172.21.15.169:0/1696006111 conn(0x7fb40c0f0be0 msgr2=0x7fb40c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:39.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 -- 172.21.15.169:0/1696006111 shutdown_connections 2024-01-11T05:09:39.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.815+0000 7fb411248700 1 -- 172.21.15.169:0/1696006111 wait complete. 2024-01-11T05:09:39.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb411248700 1 Processor -- start 2024-01-11T05:09:39.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb411248700 1 -- start start 2024-01-11T05:09:39.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb411248700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:39.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb411248700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c060320 con 0x7fb40c0f5480 2024-01-11T05:09:39.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb40ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:39.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb40ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56658/0 (socket says 172.21.15.169:56658) 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.816+0000 7fb40ad9d700 1 -- 172.21.15.169:0/1364171265 learned_addr learned my addr 172.21.15.169:0/1364171265 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb40ad9d700 1 -- 172.21.15.169:0/1364171265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4005040 con 0x7fb40c0f5480 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb40ad9d700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c1094f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fb3f40089f0 tx=0x7fb3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f4009320 con 0x7fb40c0f5480 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c060520 con 0x7fb40c0f5480 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb3f4013ec0 con 0x7fb40c0f5480 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3f4014d30 con 0x7fb40c0f5480 2024-01-11T05:09:39.818 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.817+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c109ba0 con 0x7fb40c0f5480 2024-01-11T05:09:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.818+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb3f40263a0 con 0x7fb40c0f5480 2024-01-11T05:09:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.818+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c045bb0 con 0x7fb40c0f5480 2024-01-11T05:09:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.818+0000 7fb4037fe700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb3f8031090 0x7fb3f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.818+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb3f4042390 con 0x7fb40c0f5480 2024-01-11T05:09:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.818+0000 7fb40a59c700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb3f8031090 0x7fb3f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:39.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.819+0000 7fb40a59c700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb3f8031090 0x7fb3f8033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb3fc0096a0 tx=0x7fb3fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:39.823 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:39.823+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb3f40126f0 con 0x7fb40c0f5480 2024-01-11T05:09:40.038 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.038+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb40c0faf70 con 0x7fb40c0f5480 2024-01-11T05:09:40.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.039+0000 7fb4037fe700 1 -- 172.21.15.169:0/1364171265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3f4012330 con 0x7fb40c0f5480 2024-01-11T05:09:40.039 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:40.039 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:40.040 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb3f8031090 msgr2=0x7fb3f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb3f8031090 0x7fb3f8033550 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb3fc0096a0 tx=0x7fb3fc006b40).stop 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 msgr2=0x7fb40c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c1094f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fb3f40089f0 tx=0x7fb3f400b040).stop 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 shutdown_connections 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb3f8031090 0x7fb3f8033550 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 --2- 172.21.15.169:0/1364171265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c0f5480 0x7fb40c1094f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 >> 172.21.15.169:0/1364171265 conn(0x7fb40c0f0be0 msgr2=0x7fb40c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.040+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 shutdown_connections 2024-01-11T05:09:40.041 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:40.041+0000 7fb411248700 1 -- 172.21.15.169:0/1364171265 wait complete. 2024-01-11T05:09:40.043 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:40.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:40 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:40.039075+0000 mon.smithi067 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.169:0/1364171265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:41.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:41.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:40.329189+0000 mgr.smithi067.srbzpc (mgr.14156) 197 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 -- 172.21.15.169:0/313039280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 msgr2=0x7f500c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 --2- 172.21.15.169:0/313039280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c0f58a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5000009fb0 tx=0x7f5000009b30).stop 2024-01-11T05:09:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 -- 172.21.15.169:0/313039280 shutdown_connections 2024-01-11T05:09:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 --2- 172.21.15.169:0/313039280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c0f58a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 -- 172.21.15.169:0/313039280 >> 172.21.15.169:0/313039280 conn(0x7f500c0f0be0 msgr2=0x7f500c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:42.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 -- 172.21.15.169:0/313039280 shutdown_connections 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.466+0000 7f50139fc700 1 -- 172.21.15.169:0/313039280 wait complete. 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f50139fc700 1 Processor -- start 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f50139fc700 1 -- start start 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f50139fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f50139fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c105be0 con 0x7f500c0f5480 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f5011798700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f5011798700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56676/0 (socket says 172.21.15.169:56676) 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f5011798700 1 -- 172.21.15.169:0/2318041412 learned_addr learned my addr 172.21.15.169:0/2318041412 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.467+0000 7f5011798700 1 -- 172.21.15.169:0/2318041412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5000005040 con 0x7f500c0f5480 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.468+0000 7f5011798700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c1056f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f50000089f0 tx=0x7f500000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.468+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000009320 con 0x7f500c0f5480 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.468+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c105de0 con 0x7f500c0f5480 2024-01-11T05:09:42.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.468+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5000013ec0 con 0x7f500c0f5480 2024-01-11T05:09:42.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.468+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000014d30 con 0x7f500c0f5480 2024-01-11T05:09:42.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.468+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c106270 con 0x7f500c0f5480 2024-01-11T05:09:42.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.469+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f50000095f0 con 0x7f500c0f5480 2024-01-11T05:09:42.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.469+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c045bb0 con 0x7f500c0f5480 2024-01-11T05:09:42.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.469+0000 7f4ffdffb700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4ff8031090 0x7f4ff8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:42.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.469+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5000042420 con 0x7f500c0f5480 2024-01-11T05:09:42.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.469+0000 7f5010f97700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4ff8031090 0x7f4ff8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:42.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.470+0000 7f5010f97700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4ff8031090 0x7f4ff8033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f50080096a0 tx=0x7f5008006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:42.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.474+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f500001d7f0 con 0x7f500c0f5480 2024-01-11T05:09:42.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.682+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f500c0faf70 con 0x7f500c0f5480 2024-01-11T05:09:42.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.682+0000 7f4ffdffb700 1 -- 172.21.15.169:0/2318041412 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f500000fe40 con 0x7f500c0f5480 2024-01-11T05:09:42.683 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:42.683 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4ff8031090 msgr2=0x7f4ff8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4ff8031090 0x7f4ff8033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f50080096a0 tx=0x7f5008006b40).stop 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 msgr2=0x7f500c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c1056f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f50000089f0 tx=0x7f500000b040).stop 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 shutdown_connections 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4ff8031090 0x7f4ff8033550 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 --2- 172.21.15.169:0/2318041412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f500c0f5480 0x7f500c1056f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 >> 172.21.15.169:0/2318041412 conn(0x7f500c0f0be0 msgr2=0x7f500c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 shutdown_connections 2024-01-11T05:09:42.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:42.684+0000 7f50139fc700 1 -- 172.21.15.169:0/2318041412 wait complete. 2024-01-11T05:09:42.686 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:42.329569+0000 mgr.smithi067.srbzpc (mgr.14156) 198 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:42.682634+0000 mon.smithi067 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.169:0/2318041412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:44.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:44.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:45.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- 172.21.15.169:0/1797465748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 msgr2=0x7f612c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 --2- 172.21.15.169:0/1797465748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c0f56f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f611c009fb0 tx=0x7f611c009b30).stop 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- 172.21.15.169:0/1797465748 shutdown_connections 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 --2- 172.21.15.169:0/1797465748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c0f56f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- 172.21.15.169:0/1797465748 >> 172.21.15.169:0/1797465748 conn(0x7f612c0f0a30 msgr2=0x7f612c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- 172.21.15.169:0/1797465748 shutdown_connections 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- 172.21.15.169:0/1797465748 wait complete. 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 Processor -- start 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- start start 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.191+0000 7f6130c7f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c106af0 con 0x7f612c0f52d0 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f612b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:45.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f612b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56686/0 (socket says 172.21.15.169:56686) 2024-01-11T05:09:45.193 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f612b7fe700 1 -- 172.21.15.169:0/2926674791 learned_addr learned my addr 172.21.15.169:0/2926674791 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f612b7fe700 1 -- 172.21.15.169:0/2926674791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c005040 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f612b7fe700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c106600 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f611c0089f0 tx=0x7f611c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611c009320 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c106cf0 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f611c013ec0 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.192+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611c014d30 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.193+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c107180 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.193+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f611c0095f0 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.193+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c045bb0 con 0x7f612c0f52d0 2024-01-11T05:09:45.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.193+0000 7f6113fff700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6114031090 0x7f6114033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:45.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.194+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f611c0424a0 con 0x7f612c0f52d0 2024-01-11T05:09:45.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.194+0000 7f612affd700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6114031090 0x7f6114033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:45.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.194+0000 7f612affd700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6114031090 0x7f6114033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f61200096a0 tx=0x7f6120006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:45.198 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.198+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f611c01d980 con 0x7f612c0f52d0 2024-01-11T05:09:45.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.408+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f612c108030 con 0x7f612c0f52d0 2024-01-11T05:09:45.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.408+0000 7f6113fff700 1 -- 172.21.15.169:0/2926674791 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f611c01d380 con 0x7f612c0f52d0 2024-01-11T05:09:45.409 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:45.409 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6114031090 msgr2=0x7f6114033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6114031090 0x7f6114033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f61200096a0 tx=0x7f6120006b40).stop 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 msgr2=0x7f612c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c106600 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f611c0089f0 tx=0x7f611c00b040).stop 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 shutdown_connections 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6114031090 0x7f6114033550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 --2- 172.21.15.169:0/2926674791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f612c0f52d0 0x7f612c106600 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:45.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 >> 172.21.15.169:0/2926674791 conn(0x7f612c0f0a30 msgr2=0x7f612c05fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:45.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 shutdown_connections 2024-01-11T05:09:45.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:45.410+0000 7f6130c7f700 1 -- 172.21.15.169:0/2926674791 wait complete. 2024-01-11T05:09:45.411 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:44.329899+0000 mgr.smithi067.srbzpc (mgr.14156) 199 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:46.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:46 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:45.408771+0000 mon.smithi067 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.169:0/2926674791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:46.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:46.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:47.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:46.330242+0000 mgr.smithi067.srbzpc (mgr.14156) 200 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3002779217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 msgr2=0x7f4e740f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 --2- 172.21.15.169:0/3002779217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e740f58a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f4e68009fb0 tx=0x7f4e68009b30).stop 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3002779217 shutdown_connections 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 --2- 172.21.15.169:0/3002779217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e740f58a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3002779217 >> 172.21.15.169:0/3002779217 conn(0x7f4e740f0be0 msgr2=0x7f4e740f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3002779217 shutdown_connections 2024-01-11T05:09:47.867 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3002779217 wait complete. 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 Processor -- start 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.864+0000 7f4e7c5df700 1 -- start start 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7c5df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e741094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7c5df700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74060320 con 0x7f4e740f5480 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7a37b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e741094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7a37b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e741094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:56708/0 (socket says 172.21.15.169:56708) 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7a37b700 1 -- 172.21.15.169:0/3167578392 learned_addr learned my addr 172.21.15.169:0/3167578392 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7a37b700 1 -- 172.21.15.169:0/3167578392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68005040 con 0x7f4e740f5480 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.865+0000 7f4e7a37b700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e741094f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f4e680089f0 tx=0x7f4e6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.866+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e68009320 con 0x7f4e740f5480 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.866+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74060520 con 0x7f4e740f5480 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.866+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4e68013ec0 con 0x7f4e740f5480 2024-01-11T05:09:47.868 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.866+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e68014d30 con 0x7f4e740f5480 2024-01-11T05:09:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.866+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e74109ba0 con 0x7f4e740f5480 2024-01-11T05:09:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.866+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4e680263a0 con 0x7f4e740f5480 2024-01-11T05:09:47.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.867+0000 7f4e66ffd700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e60031090 0x7f4e60033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:47.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.867+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4e680423a0 con 0x7f4e740f5480 2024-01-11T05:09:47.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.867+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74045bb0 con 0x7f4e740f5480 2024-01-11T05:09:47.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.867+0000 7f4e79b7a700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e60031090 0x7f4e60033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:47.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.867+0000 7f4e79b7a700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e60031090 0x7f4e60033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4e700096a0 tx=0x7f4e70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:47.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:47.872+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4e6801cb10 con 0x7f4e740f5480 2024-01-11T05:09:48.080 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.079+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e740fb060 con 0x7f4e740f5480 2024-01-11T05:09:48.081 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.080+0000 7f4e66ffd700 1 -- 172.21.15.169:0/3167578392 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e6802b020 con 0x7f4e740f5480 2024-01-11T05:09:48.081 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:48.081 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e60031090 msgr2=0x7f4e60033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e60031090 0x7f4e60033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4e700096a0 tx=0x7f4e70006b40).stop 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 msgr2=0x7f4e741094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e741094f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f4e680089f0 tx=0x7f4e6800b040).stop 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 shutdown_connections 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4e60031090 0x7f4e60033550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:48.083 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 --2- 172.21.15.169:0/3167578392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e740f5480 0x7f4e741094f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:48.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 >> 172.21.15.169:0/3167578392 conn(0x7f4e740f0be0 msgr2=0x7f4e740f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:48.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 shutdown_connections 2024-01-11T05:09:48.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:48.081+0000 7f4e7c5df700 1 -- 172.21.15.169:0/3167578392 wait complete. 2024-01-11T05:09:48.084 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:48.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:48 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:48.080980+0000 mon.smithi067 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.169:0/3167578392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:49.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:49.448 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:49.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:48.330599+0000 mgr.smithi067.srbzpc (mgr.14156) 201 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:50.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 -- 172.21.15.169:0/1872321642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 msgr2=0x7f24fc0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:50.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 --2- 172.21.15.169:0/1872321642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc0f58d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f24f8009fb0 tx=0x7f24f8009b30).stop 2024-01-11T05:09:50.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 -- 172.21.15.169:0/1872321642 shutdown_connections 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 --2- 172.21.15.169:0/1872321642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc0f58d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 -- 172.21.15.169:0/1872321642 >> 172.21.15.169:0/1872321642 conn(0x7f24fc0f0c10 msgr2=0x7f24fc0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 -- 172.21.15.169:0/1872321642 shutdown_connections 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.538+0000 7f250426d700 1 -- 172.21.15.169:0/1872321642 wait complete. 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f250426d700 1 Processor -- start 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f250426d700 1 -- start start 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f250426d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f250426d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc105c10 con 0x7f24fc0f54b0 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f2502009700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f2502009700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59754/0 (socket says 172.21.15.169:59754) 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f2502009700 1 -- 172.21.15.169:0/2484756252 learned_addr learned my addr 172.21.15.169:0/2484756252 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:50.541 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.539+0000 7f2502009700 1 -- 172.21.15.169:0/2484756252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8005040 con 0x7f24fc0f54b0 2024-01-11T05:09:50.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.540+0000 7f2502009700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc105720 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f24f8000c00 tx=0x7f24f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.540+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24f80095f0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.540+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc105e10 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.540+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f24f8007e50 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.540+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24f8014e70 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.540+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc1062a0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.541+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f24f80146b0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.541+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc045bb0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.541+0000 7f24f2ffd700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24e8031090 0x7f24e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.541+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f24f8041d10 con 0x7f24fc0f54b0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.541+0000 7f2501808700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24e8031090 0x7f24e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:50.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.541+0000 7f2501808700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24e8031090 0x7f24e8033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f24ec0096a0 tx=0x7f24ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:50.547 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.546+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f24f80079c0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.754+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24fc0fafa0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.755 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.754+0000 7f24f2ffd700 1 -- 172.21.15.169:0/2484756252 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24f801b2c0 con 0x7f24fc0f54b0 2024-01-11T05:09:50.756 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:50.756 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:50.757 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24e8031090 msgr2=0x7f24e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24e8031090 0x7f24e8033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f24ec0096a0 tx=0x7f24ec006b40).stop 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 msgr2=0x7f24fc105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc105720 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f24f8000c00 tx=0x7f24f800b040).stop 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 shutdown_connections 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f24e8031090 0x7f24e8033550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 --2- 172.21.15.169:0/2484756252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24fc0f54b0 0x7f24fc105720 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 >> 172.21.15.169:0/2484756252 conn(0x7f24fc0f0c10 msgr2=0x7f24fc0f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 shutdown_connections 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:50.756+0000 7f250426d700 1 -- 172.21.15.169:0/2484756252 wait complete. 2024-01-11T05:09:50.758 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:50.330928+0000 mgr.smithi067.srbzpc (mgr.14156) 202 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:50.755848+0000 mon.smithi067 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.169:0/2484756252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:52.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:52.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:53.244 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 -- 172.21.15.169:0/3126602044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 msgr2=0x7f8e880f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 --2- 172.21.15.169:0/3126602044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e880f5e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f8e7c009fb0 tx=0x7f8e7c009b30).stop 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 -- 172.21.15.169:0/3126602044 shutdown_connections 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 --2- 172.21.15.169:0/3126602044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e880f5e70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 -- 172.21.15.169:0/3126602044 >> 172.21.15.169:0/3126602044 conn(0x7f8e880f0360 msgr2=0x7f8e880f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 -- 172.21.15.169:0/3126602044 shutdown_connections 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.242+0000 7f8e8d699700 1 -- 172.21.15.169:0/3126602044 wait complete. 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.243+0000 7f8e8d699700 1 Processor -- start 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.243+0000 7f8e8d699700 1 -- start start 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e8d699700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e88109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:53.245 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e8d699700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880f5410 con 0x7f8e880f5a50 2024-01-11T05:09:53.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e86ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e88109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:53.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e86ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e88109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59768/0 (socket says 172.21.15.169:59768) 2024-01-11T05:09:53.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e86ffd700 1 -- 172.21.15.169:0/1166152775 learned_addr learned my addr 172.21.15.169:0/1166152775 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:53.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e86ffd700 1 -- 172.21.15.169:0/1166152775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c005040 con 0x7f8e880f5a50 2024-01-11T05:09:53.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e86ffd700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e88109510 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8e7c0089f0 tx=0x7f8e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:53.247 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e7c009320 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e880f5610 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e7c013ec0 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e7c014d30 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.244+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e88109bc0 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.245+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8e7c014420 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.245+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e880591e0 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.245+0000 7f8e6f7fe700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8e700310e0 0x7f8e700335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.246+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e7c0424a0 con 0x7f8e880f5a50 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.246+0000 7f8e867fc700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8e700310e0 0x7f8e700335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:53.248 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.246+0000 7f8e867fc700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8e700310e0 0x7f8e700335a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8e780096a0 tx=0x7f8e78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:53.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.250+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e7c01d7f0 con 0x7f8e880f5a50 2024-01-11T05:09:53.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.461+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e88045bb0 con 0x7f8e880f5a50 2024-01-11T05:09:53.462 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.461+0000 7f8e6f7fe700 1 -- 172.21.15.169:0/1166152775 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e7c01d7f0 con 0x7f8e880f5a50 2024-01-11T05:09:53.463 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:53.463 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8e700310e0 msgr2=0x7f8e700335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8e700310e0 0x7f8e700335a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8e780096a0 tx=0x7f8e78006b40).stop 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 msgr2=0x7f8e88109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e88109510 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8e7c0089f0 tx=0x7f8e7c00b040).stop 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 shutdown_connections 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8e700310e0 0x7f8e700335a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 --2- 172.21.15.169:0/1166152775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e880f5a50 0x7f8e88109510 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 >> 172.21.15.169:0/1166152775 conn(0x7f8e880f0360 msgr2=0x7f8e880f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 shutdown_connections 2024-01-11T05:09:53.464 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:53.463+0000 7f8e8d699700 1 -- 172.21.15.169:0/1166152775 wait complete. 2024-01-11T05:09:53.465 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:53.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:52.331310+0000 mgr.smithi067.srbzpc (mgr.14156) 203 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:54.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:54 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:53.462540+0000 mon.smithi067 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.169:0/1166152775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:54.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:54.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:55.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:54.331732+0000 mgr.smithi067.srbzpc (mgr.14156) 204 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:55.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 -- 172.21.15.169:0/2567989802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 msgr2=0x7efbf00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 --2- 172.21.15.169:0/2567989802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf00f69b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7efbe0004760 tx=0x7efbe0009ad0).stop 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 -- 172.21.15.169:0/2567989802 shutdown_connections 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 --2- 172.21.15.169:0/2567989802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf00f69b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 -- 172.21.15.169:0/2567989802 >> 172.21.15.169:0/2567989802 conn(0x7efbf00f1cd0 msgr2=0x7efbf00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 -- 172.21.15.169:0/2567989802 shutdown_connections 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.949+0000 7efbf8618700 1 -- 172.21.15.169:0/2567989802 wait complete. 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf8618700 1 Processor -- start 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf8618700 1 -- start start 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf8618700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:55.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf8618700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf0069f00 con 0x7efbf00f6590 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf63b4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf63b4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59780/0 (socket says 172.21.15.169:59780) 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf63b4700 1 -- 172.21.15.169:0/3005810864 learned_addr learned my addr 172.21.15.169:0/3005810864 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf63b4700 1 -- 172.21.15.169:0/3005810864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe0009d70 con 0x7efbf00f6590 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.950+0000 7efbf63b4700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf006ba80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7efbe0004170 tx=0x7efbe0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbe000e040 con 0x7efbf00f6590 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf006a0a0 con 0x7efbf00f6590 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efbe0008ef0 con 0x7efbf00f6590 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbe0014c90 con 0x7efbf00f6590 2024-01-11T05:09:55.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf006a530 con 0x7efbf00f6590 2024-01-11T05:09:55.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efbe0014420 con 0x7efbf00f6590 2024-01-11T05:09:55.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.951+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf00591e0 con 0x7efbf00f6590 2024-01-11T05:09:55.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.952+0000 7efbe6ffd700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efbdc031090 0x7efbdc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:55.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.952+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efbe0042260 con 0x7efbf00f6590 2024-01-11T05:09:55.954 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.952+0000 7efbf5bb3700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efbdc031090 0x7efbdc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:55.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.952+0000 7efbf5bb3700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efbdc031090 0x7efbdc033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7efbec0096a0 tx=0x7efbec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:55.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:55.956+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efbe0026300 con 0x7efbf00f6590 2024-01-11T05:09:56.167 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.166+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efbf0045bb0 con 0x7efbf00f6590 2024-01-11T05:09:56.168 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.167+0000 7efbe6ffd700 1 -- 172.21.15.169:0/3005810864 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efbe0012b20 con 0x7efbf00f6590 2024-01-11T05:09:56.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:56.168 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efbdc031090 msgr2=0x7efbdc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efbdc031090 0x7efbdc033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7efbec0096a0 tx=0x7efbec006b40).stop 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 msgr2=0x7efbf006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf006ba80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7efbe0004170 tx=0x7efbe0009ad0).stop 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 shutdown_connections 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7efbdc031090 0x7efbdc033550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 --2- 172.21.15.169:0/3005810864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efbf00f6590 0x7efbf006ba80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 >> 172.21.15.169:0/3005810864 conn(0x7efbf00f1cd0 msgr2=0x7efbf00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:56.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 shutdown_connections 2024-01-11T05:09:56.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:56.169+0000 7efbf8618700 1 -- 172.21.15.169:0/3005810864 wait complete. 2024-01-11T05:09:56.171 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:56.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:56 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:56.168461+0000 mon.smithi067 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.169:0/3005810864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:09:57.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:09:57.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:09:57.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:56.332009+0000 mgr.smithi067.srbzpc (mgr.14156) 205 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:58.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.645+0000 7f59a412a700 1 -- 172.21.15.169:0/1349109265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 msgr2=0x7f599c0f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:58.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.645+0000 7f59a412a700 1 --2- 172.21.15.169:0/1349109265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c0f5020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f598c009fb0 tx=0x7f598c009b30).stop 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.645+0000 7f59a412a700 1 -- 172.21.15.169:0/1349109265 shutdown_connections 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.645+0000 7f59a412a700 1 --2- 172.21.15.169:0/1349109265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c0f5020 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.645+0000 7f59a412a700 1 -- 172.21.15.169:0/1349109265 >> 172.21.15.169:0/1349109265 conn(0x7f599c0f0360 msgr2=0x7f599c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.646+0000 7f59a412a700 1 -- 172.21.15.169:0/1349109265 shutdown_connections 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.646+0000 7f59a412a700 1 -- 172.21.15.169:0/1349109265 wait complete. 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.646+0000 7f59a412a700 1 Processor -- start 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.646+0000 7f59a412a700 1 -- start start 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.646+0000 7f59a412a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c107f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.646+0000 7f59a412a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c108420 con 0x7f599c0f4c00 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a1ec6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:58.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a1ec6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:59804/0 (socket says 172.21.15.169:59804) 2024-01-11T05:09:58.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a1ec6700 1 -- 172.21.15.169:0/1170316152 learned_addr learned my addr 172.21.15.169:0/1170316152 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:09:58.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a1ec6700 1 -- 172.21.15.169:0/1170316152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c005040 con 0x7f599c0f4c00 2024-01-11T05:09:58.650 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a1ec6700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c107f30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f598c0089f0 tx=0x7f598c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598c009320 con 0x7f599c0f4c00 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c108620 con 0x7f599c0f4c00 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f598c013ec0 con 0x7f599c0f4c00 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598c014d30 con 0x7f599c0f4c00 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.647+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c108a30 con 0x7f599c0f4c00 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.648+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f598c014420 con 0x7f599c0f4c00 2024-01-11T05:09:58.651 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.648+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c0591e0 con 0x7f599c0f4c00 2024-01-11T05:09:58.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.648+0000 7f59927fc700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5988031090 0x7f5988033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:09:58.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.648+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f598c0424c0 con 0x7f599c0f4c00 2024-01-11T05:09:58.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.648+0000 7f59a16c5700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5988031090 0x7f5988033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:09:58.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.649+0000 7f59a16c5700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5988031090 0x7f5988033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f59980096a0 tx=0x7f5998006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:09:58.654 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.653+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f598c01d980 con 0x7f599c0f4c00 2024-01-11T05:09:58.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.861+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f599c0fa6f0 con 0x7f599c0f4c00 2024-01-11T05:09:58.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.862+0000 7f59927fc700 1 -- 172.21.15.169:0/1170316152 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f598c01d380 con 0x7f599c0f4c00 2024-01-11T05:09:58.863 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:09:58.863 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5988031090 msgr2=0x7f5988033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5988031090 0x7f5988033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f59980096a0 tx=0x7f5998006b40).stop 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 msgr2=0x7f599c107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c107f30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f598c0089f0 tx=0x7f598c00b040).stop 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 shutdown_connections 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f5988031090 0x7f5988033550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 --2- 172.21.15.169:0/1170316152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c0f4c00 0x7f599c107f30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:09:58.865 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 >> 172.21.15.169:0/1170316152 conn(0x7f599c0f0360 msgr2=0x7f599c0f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:09:58.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 shutdown_connections 2024-01-11T05:09:58.866 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:09:58.864+0000 7f59a412a700 1 -- 172.21.15.169:0/1170316152 wait complete. 2024-01-11T05:09:58.866 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:09:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:09:58.332248+0000 mgr.smithi067.srbzpc (mgr.14156) 206 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:09:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:09:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:09:58.863355+0000 mon.smithi067 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.169:0/1170316152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:00.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:00.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 -- 172.21.15.169:0/1431899814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 msgr2=0x7f6fb00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 --2- 172.21.15.169:0/1431899814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb00f58d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f6fa0009fb0 tx=0x7f6fa0009b30).stop 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 -- 172.21.15.169:0/1431899814 shutdown_connections 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 --2- 172.21.15.169:0/1431899814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb00f58d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 -- 172.21.15.169:0/1431899814 >> 172.21.15.169:0/1431899814 conn(0x7f6fb00f0c10 msgr2=0x7f6fb00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 -- 172.21.15.169:0/1431899814 shutdown_connections 2024-01-11T05:10:01.326 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.324+0000 7f6fb821e700 1 -- 172.21.15.169:0/1431899814 wait complete. 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.325+0000 7f6fb821e700 1 Processor -- start 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.325+0000 7f6fb821e700 1 -- start start 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.325+0000 7f6fb821e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb01056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.325+0000 7f6fb821e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0105b90 con 0x7f6fb00f54b0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb5fba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb01056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb5fba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb01056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51970/0 (socket says 172.21.15.169:51970) 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb5fba700 1 -- 172.21.15.169:0/2236952515 learned_addr learned my addr 172.21.15.169:0/2236952515 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb5fba700 1 -- 172.21.15.169:0/2236952515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa0005040 con 0x7f6fb00f54b0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb5fba700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb01056a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f6fa00089f0 tx=0x7f6fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fa0009320 con 0x7f6fb00f54b0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb0105d90 con 0x7f6fb00f54b0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6fa0013ec0 con 0x7f6fb00f54b0 2024-01-11T05:10:01.328 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fa0014d30 con 0x7f6fb00f54b0 2024-01-11T05:10:01.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.326+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb0106220 con 0x7f6fb00f54b0 2024-01-11T05:10:01.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.327+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6fa00095f0 con 0x7f6fb00f54b0 2024-01-11T05:10:01.329 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.327+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb0045bb0 con 0x7f6fb00f54b0 2024-01-11T05:10:01.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.327+0000 7f6fa67fc700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f9c031090 0x7f6f9c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:01.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.328+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6fa00423e0 con 0x7f6fb00f54b0 2024-01-11T05:10:01.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.328+0000 7f6fb57b9700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f9c031090 0x7f6f9c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:01.330 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.328+0000 7f6fb57b9700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f9c031090 0x7f6f9c033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6fac0096a0 tx=0x7f6fac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:01.333 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.332+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6fa001d980 con 0x7f6fb00f54b0 2024-01-11T05:10:01.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.539+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fb01070d0 con 0x7f6fb00f54b0 2024-01-11T05:10:01.540 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.539+0000 7f6fa67fc700 1 -- 172.21.15.169:0/2236952515 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fa001d380 con 0x7f6fb00f54b0 2024-01-11T05:10:01.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:01.541 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f9c031090 msgr2=0x7f6f9c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f9c031090 0x7f6f9c033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6fac0096a0 tx=0x7f6fac006b40).stop 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 msgr2=0x7f6fb01056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb01056a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f6fa00089f0 tx=0x7f6fa000b040).stop 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 shutdown_connections 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6f9c031090 0x7f6f9c033550 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:01.542 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 --2- 172.21.15.169:0/2236952515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fb00f54b0 0x7f6fb01056a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:01.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 >> 172.21.15.169:0/2236952515 conn(0x7f6fb00f0c10 msgr2=0x7f6fb00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:01.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 shutdown_connections 2024-01-11T05:10:01.543 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:01.541+0000 7f6fb821e700 1 -- 172.21.15.169:0/2236952515 wait complete. 2024-01-11T05:10:01.543 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:00.332634+0000 mgr.smithi067.srbzpc (mgr.14156) 207 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:02.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:02 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:01.540654+0000 mon.smithi067 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.169:0/2236952515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:02.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:02.921 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:02.333071+0000 mgr.smithi067.srbzpc (mgr.14156) 208 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:04.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 -- 172.21.15.169:0/4258189000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 msgr2=0x7f85e80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:04.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 --2- 172.21.15.169:0/4258189000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e80f58a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f85dc009fb0 tx=0x7f85dc009b30).stop 2024-01-11T05:10:04.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 -- 172.21.15.169:0/4258189000 shutdown_connections 2024-01-11T05:10:04.055 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 --2- 172.21.15.169:0/4258189000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e80f58a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 -- 172.21.15.169:0/4258189000 >> 172.21.15.169:0/4258189000 conn(0x7f85e80f0be0 msgr2=0x7f85e80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 -- 172.21.15.169:0/4258189000 shutdown_connections 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.053+0000 7f85f00e7700 1 -- 172.21.15.169:0/4258189000 wait complete. 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85f00e7700 1 Processor -- start 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85f00e7700 1 -- start start 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85f00e7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85f00e7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e8060300 con 0x7f85e80f5480 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85ede83700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85ede83700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51986/0 (socket says 172.21.15.169:51986) 2024-01-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85ede83700 1 -- 172.21.15.169:0/2721943624 learned_addr learned my addr 172.21.15.169:0/2721943624 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85ede83700 1 -- 172.21.15.169:0/2721943624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc005040 con 0x7f85e80f5480 2024-01-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.054+0000 7f85ede83700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e8109510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f85dc0089f0 tx=0x7f85dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.055+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc009320 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.055+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e8060500 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.055+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f85dc013ec0 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.055+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc014d30 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.055+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e8109bc0 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.055+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f85dc0095f0 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.056+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e80591e0 con 0x7f85e80f5480 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.056+0000 7f85da7fc700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f85d4031090 0x7f85d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:04.058 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.056+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f85dc042460 con 0x7f85e80f5480 2024-01-11T05:10:04.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.056+0000 7f85ed682700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f85d4031090 0x7f85d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:04.059 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.056+0000 7f85ed682700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f85d4031090 0x7f85d4033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f85e40096a0 tx=0x7f85e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:04.061 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.061+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f85dc01d7f0 con 0x7f85e80f5480 2024-01-11T05:10:04.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.271+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85e8045bb0 con 0x7f85e80f5480 2024-01-11T05:10:04.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.271+0000 7f85da7fc700 1 -- 172.21.15.169:0/2721943624 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85dc01d7f0 con 0x7f85e80f5480 2024-01-11T05:10:04.273 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:04.273 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f85d4031090 msgr2=0x7f85d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f85d4031090 0x7f85d4033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f85e40096a0 tx=0x7f85e4006b40).stop 2024-01-11T05:10:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 msgr2=0x7f85e8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e8109510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f85dc0089f0 tx=0x7f85dc00b040).stop 2024-01-11T05:10:04.274 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 shutdown_connections 2024-01-11T05:10:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f85d4031090 0x7f85d4033550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 --2- 172.21.15.169:0/2721943624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e80f5480 0x7f85e8109510 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 >> 172.21.15.169:0/2721943624 conn(0x7f85e80f0be0 msgr2=0x7f85e80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 shutdown_connections 2024-01-11T05:10:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:04.273+0000 7f85f00e7700 1 -- 172.21.15.169:0/2721943624 wait complete. 2024-01-11T05:10:04.275 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:04.272828+0000 mon.smithi067 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.169:0/2721943624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:05.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:04.333464+0000 mgr.smithi067.srbzpc (mgr.14156) 209 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:05.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:05.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:06.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 -- 172.21.15.169:0/3515202215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 msgr2=0x7faa000f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:06.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 --2- 172.21.15.169:0/3515202215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa000f69b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fa9f0004760 tx=0x7fa9f0009ad0).stop 2024-01-11T05:10:06.797 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 -- 172.21.15.169:0/3515202215 shutdown_connections 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 --2- 172.21.15.169:0/3515202215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa000f69b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 -- 172.21.15.169:0/3515202215 >> 172.21.15.169:0/3515202215 conn(0x7faa000f1cd0 msgr2=0x7faa000f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 -- 172.21.15.169:0/3515202215 shutdown_connections 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.794+0000 7faa07653700 1 -- 172.21.15.169:0/3515202215 wait complete. 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa07653700 1 Processor -- start 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa07653700 1 -- start start 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa07653700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa001067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa07653700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00106cb0 con 0x7faa000f6590 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa053ef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa001067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa053ef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa001067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:51990/0 (socket says 172.21.15.169:51990) 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa053ef700 1 -- 172.21.15.169:0/1577453063 learned_addr learned my addr 172.21.15.169:0/1577453063 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.795+0000 7faa053ef700 1 -- 172.21.15.169:0/1577453063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0009d70 con 0x7faa000f6590 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.796+0000 7faa053ef700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa001067c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fa9f0013f10 tx=0x7fa9f0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:06.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.796+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f000e040 con 0x7faa000f6590 2024-01-11T05:10:06.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.796+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00106eb0 con 0x7faa000f6590 2024-01-11T05:10:06.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.796+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa9f0007df0 con 0x7faa000f6590 2024-01-11T05:10:06.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.796+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f0014d60 con 0x7faa000f6590 2024-01-11T05:10:06.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.796+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa00107340 con 0x7faa000f6590 2024-01-11T05:10:06.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.797+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa9f0014520 con 0x7faa000f6590 2024-01-11T05:10:06.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.797+0000 7fa9f5ffb700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa9ec031090 0x7fa9ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:06.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.797+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa00045bb0 con 0x7faa000f6590 2024-01-11T05:10:06.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.797+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa9f00424c0 con 0x7faa000f6590 2024-01-11T05:10:06.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.797+0000 7faa04bee700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa9ec031090 0x7fa9ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:06.800 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.798+0000 7faa04bee700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa9ec031090 0x7fa9ec033550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa9fc0096a0 tx=0x7fa9fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:06.802 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:06.802+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa9f0007960 con 0x7faa000f6590 2024-01-11T05:10:07.009 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.008+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa00108160 con 0x7faa000f6590 2024-01-11T05:10:07.010 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.009+0000 7fa9f5ffb700 1 -- 172.21.15.169:0/1577453063 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9f00262f0 con 0x7faa000f6590 2024-01-11T05:10:07.010 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:07.010 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa9ec031090 msgr2=0x7fa9ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa9ec031090 0x7fa9ec033550 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa9fc0096a0 tx=0x7fa9fc006b40).stop 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 msgr2=0x7faa001067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa001067c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fa9f0013f10 tx=0x7fa9f0009ad0).stop 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 shutdown_connections 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa9ec031090 0x7fa9ec033550 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 --2- 172.21.15.169:0/1577453063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa000f6590 0x7faa001067c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 >> 172.21.15.169:0/1577453063 conn(0x7faa000f1cd0 msgr2=0x7faa000f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 shutdown_connections 2024-01-11T05:10:07.012 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:07.011+0000 7faa07653700 1 -- 172.21.15.169:0/1577453063 wait complete. 2024-01-11T05:10:07.013 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:07.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:06.333908+0000 mgr.smithi067.srbzpc (mgr.14156) 210 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:07.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:07.010283+0000 mon.smithi067 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.169:0/1577453063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:08.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:08.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:09.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 -- 172.21.15.169:0/1555147374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 msgr2=0x7ff6040f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:09.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 --2- 172.21.15.169:0/1555147374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff6040f56f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff600009fb0 tx=0x7ff600009b30).stop 2024-01-11T05:10:09.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 -- 172.21.15.169:0/1555147374 shutdown_connections 2024-01-11T05:10:09.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 --2- 172.21.15.169:0/1555147374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff6040f56f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:09.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 -- 172.21.15.169:0/1555147374 >> 172.21.15.169:0/1555147374 conn(0x7ff6040f0a30 msgr2=0x7ff6040f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:09.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 -- 172.21.15.169:0/1555147374 shutdown_connections 2024-01-11T05:10:09.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.548+0000 7ff60a1cd700 1 -- 172.21.15.169:0/1555147374 wait complete. 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff60a1cd700 1 Processor -- start 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff60a1cd700 1 -- start start 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff60a1cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff604109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff60a1cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff604060300 con 0x7ff6040f52d0 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff6091cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff604109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff6091cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff604109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43508/0 (socket says 172.21.15.169:43508) 2024-01-11T05:10:09.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff6091cb700 1 -- 172.21.15.169:0/2845875131 learned_addr learned my addr 172.21.15.169:0/2845875131 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.549+0000 7ff6091cb700 1 -- 172.21.15.169:0/2845875131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600005040 con 0x7ff6040f52d0 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.550+0000 7ff6091cb700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff604109330 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7ff6000089f0 tx=0x7ff60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.550+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff600009320 con 0x7ff6040f52d0 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.550+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff604060500 con 0x7ff6040f52d0 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.550+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff600013ec0 con 0x7ff6040f52d0 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.550+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff600014d30 con 0x7ff6040f52d0 2024-01-11T05:10:09.552 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.550+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6041099e0 con 0x7ff6040f52d0 2024-01-11T05:10:09.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.551+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff6000095f0 con 0x7ff6040f52d0 2024-01-11T05:10:09.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.551+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6040591e0 con 0x7ff6040f52d0 2024-01-11T05:10:09.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.551+0000 7ff5f9ffb700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5f0031090 0x7ff5f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:09.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.551+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff6000424a0 con 0x7ff6040f52d0 2024-01-11T05:10:09.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.551+0000 7ff6089ca700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5f0031090 0x7ff5f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:09.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.552+0000 7ff6089ca700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5f0031090 0x7ff5f0033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff5f40096a0 tx=0x7ff5f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:09.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.556+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff60001d7f0 con 0x7ff6040f52d0 2024-01-11T05:10:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:08.334334+0000 mgr.smithi067.srbzpc (mgr.14156) 211 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:09.763 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.762+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff604045bb0 con 0x7ff6040f52d0 2024-01-11T05:10:09.764 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.763+0000 7ff5f9ffb700 1 -- 172.21.15.169:0/2845875131 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff60001d7f0 con 0x7ff6040f52d0 2024-01-11T05:10:09.764 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:09.764 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:09.765 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5f0031090 msgr2=0x7ff5f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5f0031090 0x7ff5f0033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff5f40096a0 tx=0x7ff5f4006b40).stop 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 msgr2=0x7ff604109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff604109330 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7ff6000089f0 tx=0x7ff60000b040).stop 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 shutdown_connections 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ff5f0031090 0x7ff5f0033550 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 --2- 172.21.15.169:0/2845875131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6040f52d0 0x7ff604109330 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.764+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 >> 172.21.15.169:0/2845875131 conn(0x7ff6040f0a30 msgr2=0x7ff6040f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.765+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 shutdown_connections 2024-01-11T05:10:09.766 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:09.765+0000 7ff60a1cd700 1 -- 172.21.15.169:0/2845875131 wait complete. 2024-01-11T05:10:09.767 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:10.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:10 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:09.764318+0000 mon.smithi067 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.169:0/2845875131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:11.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:11.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:10.334640+0000 mgr.smithi067.srbzpc (mgr.14156) 212 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:12.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.266+0000 7f6348ea9700 1 -- 172.21.15.169:0/2919749226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 msgr2=0x7f63440f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.266+0000 7f6348ea9700 1 --2- 172.21.15.169:0/2919749226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f63440f56f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f6334009fb0 tx=0x7f6334009b30).stop 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.266+0000 7f6348ea9700 1 -- 172.21.15.169:0/2919749226 shutdown_connections 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.266+0000 7f6348ea9700 1 --2- 172.21.15.169:0/2919749226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f63440f56f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.266+0000 7f6348ea9700 1 -- 172.21.15.169:0/2919749226 >> 172.21.15.169:0/2919749226 conn(0x7f63440f0a30 msgr2=0x7f63440f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.266+0000 7f6348ea9700 1 -- 172.21.15.169:0/2919749226 shutdown_connections 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.267+0000 7f6348ea9700 1 -- 172.21.15.169:0/2919749226 wait complete. 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.267+0000 7f6348ea9700 1 Processor -- start 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.267+0000 7f6348ea9700 1 -- start start 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.267+0000 7f6348ea9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f6344109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.267+0000 7f6348ea9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6344060320 con 0x7f63440f52d0 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f63437fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f6344109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:12.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f63437fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f6344109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43528/0 (socket says 172.21.15.169:43528) 2024-01-11T05:10:12.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f63437fe700 1 -- 172.21.15.169:0/802327641 learned_addr learned my addr 172.21.15.169:0/802327641 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:12.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f63437fe700 1 -- 172.21.15.169:0/802327641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334005040 con 0x7f63440f52d0 2024-01-11T05:10:12.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f63437fe700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f6344109310 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f63340089f0 tx=0x7f633400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:12.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6334009320 con 0x7f63440f52d0 2024-01-11T05:10:12.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6344060520 con 0x7f63440f52d0 2024-01-11T05:10:12.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6334013ec0 con 0x7f63440f52d0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6334014d30 con 0x7f63440f52d0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.268+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63441099c0 con 0x7f63440f52d0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.269+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6334014420 con 0x7f63440f52d0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.269+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6344045bb0 con 0x7f63440f52d0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.269+0000 7f632bfff700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f632c031090 0x7f632c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.269+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f63340423e0 con 0x7f63440f52d0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.269+0000 7f6342ffd700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f632c031090 0x7f632c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:12.272 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.270+0000 7f6342ffd700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f632c031090 0x7f632c033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f63380096a0 tx=0x7f6338006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:12.275 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.274+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f63340268c0 con 0x7f63440f52d0 2024-01-11T05:10:12.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.480+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f634410a1e0 con 0x7f63440f52d0 2024-01-11T05:10:12.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.481+0000 7f632bfff700 1 -- 172.21.15.169:0/802327641 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f633401d7f0 con 0x7f63440f52d0 2024-01-11T05:10:12.482 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:12.482 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f632c031090 msgr2=0x7f632c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f632c031090 0x7f632c033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f63380096a0 tx=0x7f6338006b40).stop 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 msgr2=0x7f6344109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f6344109310 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f63340089f0 tx=0x7f633400b040).stop 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 shutdown_connections 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f632c031090 0x7f632c033550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 --2- 172.21.15.169:0/802327641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63440f52d0 0x7f6344109310 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 >> 172.21.15.169:0/802327641 conn(0x7f63440f0a30 msgr2=0x7f63440f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 shutdown_connections 2024-01-11T05:10:12.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:12.482+0000 7f6348ea9700 1 -- 172.21.15.169:0/802327641 wait complete. 2024-01-11T05:10:12.485 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:12.334894+0000 mgr.smithi067.srbzpc (mgr.14156) 213 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:12.482115+0000 mon.smithi067 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.169:0/802327641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:13.849 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:13.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:14.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 -- 172.21.15.169:0/190630101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 msgr2=0x7f2ee40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:14.960 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 --2- 172.21.15.169:0/190630101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee40f69b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2ed8009fb0 tx=0x7f2ed8009b30).stop 2024-01-11T05:10:14.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 -- 172.21.15.169:0/190630101 shutdown_connections 2024-01-11T05:10:14.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 --2- 172.21.15.169:0/190630101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee40f69b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:14.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 -- 172.21.15.169:0/190630101 >> 172.21.15.169:0/190630101 conn(0x7f2ee40f1d10 msgr2=0x7f2ee40f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:14.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 -- 172.21.15.169:0/190630101 shutdown_connections 2024-01-11T05:10:14.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.955+0000 7f2ee8af4700 1 -- 172.21.15.169:0/190630101 wait complete. 2024-01-11T05:10:14.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee8af4700 1 Processor -- start 2024-01-11T05:10:14.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee8af4700 1 -- start start 2024-01-11T05:10:14.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee8af4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:14.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee8af4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee4069f00 con 0x7f2ee40f6590 2024-01-11T05:10:14.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:14.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43546/0 (socket says 172.21.15.169:43546) 2024-01-11T05:10:14.962 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.956+0000 7f2ee259c700 1 -- 172.21.15.169:0/1599028265 learned_addr learned my addr 172.21.15.169:0/1599028265 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ee259c700 1 -- 172.21.15.169:0/1599028265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8005040 con 0x7f2ee40f6590 2024-01-11T05:10:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ee259c700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee406ba80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f2ed80089f0 tx=0x7f2ed800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed8009320 con 0x7f2ee40f6590 2024-01-11T05:10:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee406a0a0 con 0x7f2ee40f6590 2024-01-11T05:10:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2ed8013ec0 con 0x7f2ee40f6590 2024-01-11T05:10:14.963 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed8014d30 con 0x7f2ee40f6590 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.957+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee406a530 con 0x7f2ee40f6590 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.958+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2ed80095f0 con 0x7f2ee40f6590 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.958+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee40591e0 con 0x7f2ee40f6590 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.958+0000 7f2ed2ffd700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2ecc031090 0x7f2ecc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.958+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2ed80424a0 con 0x7f2ee40f6590 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.958+0000 7f2ee1d9b700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2ecc031090 0x7f2ecc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:14.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.959+0000 7f2ee1d9b700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2ecc031090 0x7f2ecc033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2ed40096a0 tx=0x7f2ed4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:14.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:14.963+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2ed801d980 con 0x7f2ee40f6590 2024-01-11T05:10:15.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.172+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ee40fc080 con 0x7f2ee40f6590 2024-01-11T05:10:15.173 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.172+0000 7f2ed2ffd700 1 -- 172.21.15.169:0/1599028265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ed801d380 con 0x7f2ee40f6590 2024-01-11T05:10:15.173 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:15.173 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:15.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2ecc031090 msgr2=0x7f2ecc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:15.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2ecc031090 0x7f2ecc033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f2ed40096a0 tx=0x7f2ed4006b40).stop 2024-01-11T05:10:15.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 msgr2=0x7f2ee406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:15.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee406ba80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f2ed80089f0 tx=0x7f2ed800b040).stop 2024-01-11T05:10:15.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 shutdown_connections 2024-01-11T05:10:15.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2ecc031090 0x7f2ecc033550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:15.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 --2- 172.21.15.169:0/1599028265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ee40f6590 0x7f2ee406ba80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:15.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 >> 172.21.15.169:0/1599028265 conn(0x7f2ee40f1d10 msgr2=0x7f2ee40f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:15.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 shutdown_connections 2024-01-11T05:10:15.176 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:15.174+0000 7f2ee8af4700 1 -- 172.21.15.169:0/1599028265 wait complete. 2024-01-11T05:10:15.176 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:15.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:14.335168+0000 mgr.smithi067.srbzpc (mgr.14156) 214 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:16.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:16 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:15.173666+0000 mon.smithi067 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.169:0/1599028265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:16.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:16.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:17.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:16.335541+0000 mgr.smithi067.srbzpc (mgr.14156) 215 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 -- 172.21.15.169:0/3178638881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 msgr2=0x7f2f200f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 --2- 172.21.15.169:0/3178638881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f200f56f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2f1c009fb0 tx=0x7f2f1c009b30).stop 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 -- 172.21.15.169:0/3178638881 shutdown_connections 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 --2- 172.21.15.169:0/3178638881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f200f56f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 -- 172.21.15.169:0/3178638881 >> 172.21.15.169:0/3178638881 conn(0x7f2f200f0a30 msgr2=0x7f2f200f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 -- 172.21.15.169:0/3178638881 shutdown_connections 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 -- 172.21.15.169:0/3178638881 wait complete. 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.689+0000 7f2f267b7700 1 Processor -- start 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f267b7700 1 -- start start 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f267b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f2010b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:17.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f267b7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20107120 con 0x7f2f200f52d0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f257b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f2010b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f257b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f2010b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:43568/0 (socket says 172.21.15.169:43568) 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f257b5700 1 -- 172.21.15.169:0/3286568306 learned_addr learned my addr 172.21.15.169:0/3286568306 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f257b5700 1 -- 172.21.15.169:0/3286568306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c005040 con 0x7f2f200f52d0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f257b5700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f2010b730 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2f1c000c00 tx=0x7f2f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f1c0095f0 con 0x7f2f200f52d0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f20060300 con 0x7f2f200f52d0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2f1c007e50 con 0x7f2f200f52d0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.690+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f1c014e70 con 0x7f2f200f52d0 2024-01-11T05:10:17.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.691+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2010bc20 con 0x7f2f200f52d0 2024-01-11T05:10:17.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.691+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2f1c025070 con 0x7f2f200f52d0 2024-01-11T05:10:17.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.692+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f20045bb0 con 0x7f2f200f52d0 2024-01-11T05:10:17.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.692+0000 7f2f15ffb700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2f0c031090 0x7f2f0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:17.694 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.692+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2f1c041d10 con 0x7f2f200f52d0 2024-01-11T05:10:17.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.692+0000 7f2f24fb4700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2f0c031090 0x7f2f0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:17.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.692+0000 7f2f24fb4700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2f0c031090 0x7f2f0c033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2f100096a0 tx=0x7f2f10006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:17.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.696+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2f1c00f4e0 con 0x7f2f200f52d0 2024-01-11T05:10:17.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.906+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f2010c3e0 con 0x7f2f200f52d0 2024-01-11T05:10:17.907 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.906+0000 7f2f15ffb700 1 -- 172.21.15.169:0/3286568306 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f1c00f300 con 0x7f2f200f52d0 2024-01-11T05:10:17.907 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:17.907 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2f0c031090 msgr2=0x7f2f0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2f0c031090 0x7f2f0c033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f2f100096a0 tx=0x7f2f10006b40).stop 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 msgr2=0x7f2f2010b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f2010b730 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2f1c000c00 tx=0x7f2f1c00b040).stop 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 shutdown_connections 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2f0c031090 0x7f2f0c033550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:17.909 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 --2- 172.21.15.169:0/3286568306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f200f52d0 0x7f2f2010b730 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:17.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 >> 172.21.15.169:0/3286568306 conn(0x7f2f200f0a30 msgr2=0x7f2f200f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:17.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 shutdown_connections 2024-01-11T05:10:17.910 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:17.908+0000 7f2f267b7700 1 -- 172.21.15.169:0/3286568306 wait complete. 2024-01-11T05:10:17.910 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:18.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:18 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:17.907632+0000 mon.smithi067 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.169:0/3286568306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:19.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:19.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:19.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:18.335955+0000 mgr.smithi067.srbzpc (mgr.14156) 216 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- 172.21.15.169:0/1554504787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 msgr2=0x7fd98c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 --2- 172.21.15.169:0/1554504787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c0f56f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fd988009fb0 tx=0x7fd988009b30).stop 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- 172.21.15.169:0/1554504787 shutdown_connections 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 --2- 172.21.15.169:0/1554504787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c0f56f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- 172.21.15.169:0/1554504787 >> 172.21.15.169:0/1554504787 conn(0x7fd98c0f0a30 msgr2=0x7fd98c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- 172.21.15.169:0/1554504787 shutdown_connections 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- 172.21.15.169:0/1554504787 wait complete. 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 Processor -- start 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- start start 2024-01-11T05:10:20.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c1092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:20.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.346+0000 7fd9926b4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c060360 con 0x7fd98c0f52d0 2024-01-11T05:10:20.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd9916b2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c1092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:20.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd9916b2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c1092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49440/0 (socket says 172.21.15.169:49440) 2024-01-11T05:10:20.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd9916b2700 1 -- 172.21.15.169:0/360864310 learned_addr learned my addr 172.21.15.169:0/360864310 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:20.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd9916b2700 1 -- 172.21.15.169:0/360864310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd988005040 con 0x7fd98c0f52d0 2024-01-11T05:10:20.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd9916b2700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c1092d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd9880089f0 tx=0x7fd98800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:20.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd988009320 con 0x7fd98c0f52d0 2024-01-11T05:10:20.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98c060560 con 0x7fd98c0f52d0 2024-01-11T05:10:20.350 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd988013ec0 con 0x7fd98c0f52d0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.347+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd988014d30 con 0x7fd98c0f52d0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.348+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98c109a00 con 0x7fd98c0f52d0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.349+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd9880095f0 con 0x7fd98c0f52d0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.349+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98c045bb0 con 0x7fd98c0f52d0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.349+0000 7fd981ffb700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd978031090 0x7fd978033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.349+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd988042420 con 0x7fd98c0f52d0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.349+0000 7fd990eb1700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd978031090 0x7fd978033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:20.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.349+0000 7fd990eb1700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd978031090 0x7fd978033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd97c0096a0 tx=0x7fd97c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:20.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.353+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd98801d7f0 con 0x7fd98c0f52d0 2024-01-11T05:10:20.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.561+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd98c0fadc0 con 0x7fd98c0f52d0 2024-01-11T05:10:20.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.562+0000 7fd981ffb700 1 -- 172.21.15.169:0/360864310 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd98800fe40 con 0x7fd98c0f52d0 2024-01-11T05:10:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:20.564 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:20.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd978031090 msgr2=0x7fd978033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:20.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd978031090 0x7fd978033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd97c0096a0 tx=0x7fd97c006b40).stop 2024-01-11T05:10:20.565 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 msgr2=0x7fd98c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c1092d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd9880089f0 tx=0x7fd98800b040).stop 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 shutdown_connections 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd978031090 0x7fd978033550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 --2- 172.21.15.169:0/360864310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd98c0f52d0 0x7fd98c1092d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 >> 172.21.15.169:0/360864310 conn(0x7fd98c0f0a30 msgr2=0x7fd98c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 shutdown_connections 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:20.564+0000 7fd9926b4700 1 -- 172.21.15.169:0/360864310 wait complete. 2024-01-11T05:10:20.566 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:20.336277+0000 mgr.smithi067.srbzpc (mgr.14156) 217 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:20.563707+0000 mon.smithi067 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.169:0/360864310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:21.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:21.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 -- 172.21.15.169:0/2498147348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 msgr2=0x7f20b80fb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 --2- 172.21.15.169:0/2498147348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b80fb5c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f20a8009fb0 tx=0x7f20a8009b30).stop 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 -- 172.21.15.169:0/2498147348 shutdown_connections 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 --2- 172.21.15.169:0/2498147348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b80fb5c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 -- 172.21.15.169:0/2498147348 >> 172.21.15.169:0/2498147348 conn(0x7f20b80f0be0 msgr2=0x7f20b80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 -- 172.21.15.169:0/2498147348 shutdown_connections 2024-01-11T05:10:23.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.082+0000 7f20bf3d6700 1 -- 172.21.15.169:0/2498147348 wait complete. 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bf3d6700 1 Processor -- start 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bf3d6700 1 -- start start 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bf3d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b8108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bf3d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b8108e60 con 0x7f20b8060300 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bd172700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b8108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bd172700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b8108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49460/0 (socket says 172.21.15.169:49460) 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bd172700 1 -- 172.21.15.169:0/424628996 learned_addr learned my addr 172.21.15.169:0/424628996 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bd172700 1 -- 172.21.15.169:0/424628996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8005040 con 0x7f20b8060300 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.083+0000 7f20bd172700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b8108970 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f20a8009f80 tx=0x7f20a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.084+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a8009320 con 0x7f20b8060300 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.084+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b8109060 con 0x7f20b8060300 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.084+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f20a8013680 con 0x7f20b8060300 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.084+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a801eda0 con 0x7f20b8060300 2024-01-11T05:10:23.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.084+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b81094f0 con 0x7f20b8060300 2024-01-11T05:10:23.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.085+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f20a801e420 con 0x7f20b8060300 2024-01-11T05:10:23.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.085+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b8045bb0 con 0x7f20b8060300 2024-01-11T05:10:23.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.085+0000 7f20adffb700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f20a4031090 0x7f20a4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:23.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.085+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f20a8042410 con 0x7f20b8060300 2024-01-11T05:10:23.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.085+0000 7f20bc971700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f20a4031090 0x7f20a4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:23.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.086+0000 7f20bc971700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f20a4031090 0x7f20a4033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f20b40096a0 tx=0x7f20b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:23.091 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.091+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f20a801d380 con 0x7f20b8060300 2024-01-11T05:10:23.298 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.297+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20b810a3a0 con 0x7f20b8060300 2024-01-11T05:10:23.299 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.299+0000 7f20adffb700 1 -- 172.21.15.169:0/424628996 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20a8004490 con 0x7f20b8060300 2024-01-11T05:10:23.299 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:23.299 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f20a4031090 msgr2=0x7f20a4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f20a4031090 0x7f20a4033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f20b40096a0 tx=0x7f20b4006b40).stop 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 msgr2=0x7f20b8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b8108970 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f20a8009f80 tx=0x7f20a800b040).stop 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 shutdown_connections 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f20a4031090 0x7f20a4033550 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:23.301 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 --2- 172.21.15.169:0/424628996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20b8060300 0x7f20b8108970 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 >> 172.21.15.169:0/424628996 conn(0x7f20b80f0be0 msgr2=0x7f20b80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 shutdown_connections 2024-01-11T05:10:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:23.300+0000 7f20bf3d6700 1 -- 172.21.15.169:0/424628996 wait complete. 2024-01-11T05:10:23.302 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:22.336628+0000 mgr.smithi067.srbzpc (mgr.14156) 218 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:22.418410+0000 mon.smithi067 (mon.0) 276 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:10:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:22.418646+0000 mon.smithi067 (mon.0) 277 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:10:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:23.299353+0000 mon.smithi067 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.169:0/424628996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:24.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:24.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:24.336986+0000 mgr.smithi067.srbzpc (mgr.14156) 219 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:25.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 -- 172.21.15.169:0/1381720264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 msgr2=0x7f42280f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:25.798 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 --2- 172.21.15.169:0/1381720264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42280f69b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f421c004760 tx=0x7f421c009ad0).stop 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 -- 172.21.15.169:0/1381720264 shutdown_connections 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 --2- 172.21.15.169:0/1381720264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42280f69b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 -- 172.21.15.169:0/1381720264 >> 172.21.15.169:0/1381720264 conn(0x7f42280f1cd0 msgr2=0x7f42280f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 -- 172.21.15.169:0/1381720264 shutdown_connections 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.796+0000 7f422eaa2700 1 -- 172.21.15.169:0/1381720264 wait complete. 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422eaa2700 1 Processor -- start 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422eaa2700 1 -- start start 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422eaa2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42281067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422eaa2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4228106cb0 con 0x7f42280f6590 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422c83e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42281067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422c83e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42281067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49474/0 (socket says 172.21.15.169:49474) 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.797+0000 7f422c83e700 1 -- 172.21.15.169:0/1007066341 learned_addr learned my addr 172.21.15.169:0/1007066341 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f422c83e700 1 -- 172.21.15.169:0/1007066341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c009d70 con 0x7f42280f6590 2024-01-11T05:10:25.799 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f422c83e700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42281067c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f421c004170 tx=0x7f421c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421c00e040 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4228106eb0 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f421c008ef0 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421c014c90 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.798+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4228107340 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.799+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f421c014420 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.799+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4228045bb0 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.799+0000 7f42257fa700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4218031090 0x7f4218033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.799+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f421c0421f0 con 0x7f42280f6590 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.799+0000 7f4227fff700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4218031090 0x7f4218033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:25.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.800+0000 7f4227fff700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4218031090 0x7f4218033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f42140096a0 tx=0x7f4214006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:25.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:25.805+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f421c026300 con 0x7f42280f6590 2024-01-11T05:10:26.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.016+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42281081f0 con 0x7f42280f6590 2024-01-11T05:10:26.017 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.016+0000 7f42257fa700 1 -- 172.21.15.169:0/1007066341 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f421c012b20 con 0x7f42280f6590 2024-01-11T05:10:26.017 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:26.018 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4218031090 msgr2=0x7f4218033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4218031090 0x7f4218033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f42140096a0 tx=0x7f4214006b40).stop 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 msgr2=0x7f42281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42281067c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f421c004170 tx=0x7f421c009ad0).stop 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 shutdown_connections 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4218031090 0x7f4218033550 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 --2- 172.21.15.169:0/1007066341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280f6590 0x7f42281067c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 >> 172.21.15.169:0/1007066341 conn(0x7f42280f1cd0 msgr2=0x7f42280f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 shutdown_connections 2024-01-11T05:10:26.019 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:26.019+0000 7f422eaa2700 1 -- 172.21.15.169:0/1007066341 wait complete. 2024-01-11T05:10:26.020 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:26.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:26 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:26.017662+0000 mon.smithi067 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.169:0/1007066341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:27.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:27.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:27.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:26.337318+0000 mgr.smithi067.srbzpc (mgr.14156) 220 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:28.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 -- 172.21.15.169:0/1280292613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 msgr2=0x7fe54c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:28.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 --2- 172.21.15.169:0/1280292613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c0f56f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fe540009fb0 tx=0x7fe540009b30).stop 2024-01-11T05:10:28.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 -- 172.21.15.169:0/1280292613 shutdown_connections 2024-01-11T05:10:28.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 --2- 172.21.15.169:0/1280292613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c0f56f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:28.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 -- 172.21.15.169:0/1280292613 >> 172.21.15.169:0/1280292613 conn(0x7fe54c0f0a30 msgr2=0x7fe54c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:28.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 -- 172.21.15.169:0/1280292613 shutdown_connections 2024-01-11T05:10:28.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.488+0000 7fe54b59e700 1 -- 172.21.15.169:0/1280292613 wait complete. 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54b59e700 1 Processor -- start 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54b59e700 1 -- start start 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54b59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54b59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c108d20 con 0x7fe54c0f52d0 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:49492/0 (socket says 172.21.15.169:49492) 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54a59c700 1 -- 172.21.15.169:0/3900161133 learned_addr learned my addr 172.21.15.169:0/3900161133 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:28.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54a59c700 1 -- 172.21.15.169:0/3900161133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe540005040 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.489+0000 7fe54a59c700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c108830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fe5400089f0 tx=0x7fe54000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.490+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe540009320 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.490+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c108f20 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.490+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe540013ec0 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.490+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe540014d30 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.490+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c1093b0 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.490+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe5400095f0 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.491+0000 7fe53affd700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe534031090 0x7fe534033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.491+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe540042400 con 0x7fe54c0f52d0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.491+0000 7fe549d9b700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe534031090 0x7fe534033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:28.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.491+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c0591e0 con 0x7fe54c0f52d0 2024-01-11T05:10:28.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.491+0000 7fe549d9b700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe534031090 0x7fe534033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe53c0096a0 tx=0x7fe53c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:28.496 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.496+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe54001ddb0 con 0x7fe54c0f52d0 2024-01-11T05:10:28.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.709+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe54c045bb0 con 0x7fe54c0f52d0 2024-01-11T05:10:28.710 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.710+0000 7fe53affd700 1 -- 172.21.15.169:0/3900161133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe54002b020 con 0x7fe54c0f52d0 2024-01-11T05:10:28.710 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:28.710 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe534031090 msgr2=0x7fe534033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe534031090 0x7fe534033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe53c0096a0 tx=0x7fe53c006b40).stop 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 msgr2=0x7fe54c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c108830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fe5400089f0 tx=0x7fe54000b040).stop 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 shutdown_connections 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe534031090 0x7fe534033550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 --2- 172.21.15.169:0/3900161133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54c0f52d0 0x7fe54c108830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 >> 172.21.15.169:0/3900161133 conn(0x7fe54c0f0a30 msgr2=0x7fe54c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 shutdown_connections 2024-01-11T05:10:28.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:28.711+0000 7fe54b59e700 1 -- 172.21.15.169:0/3900161133 wait complete. 2024-01-11T05:10:28.713 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:28.337759+0000 mgr.smithi067.srbzpc (mgr.14156) 221 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:28.709892+0000 mon.smithi067 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.169:0/3900161133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:30.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:30.075 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 -- 172.21.15.169:0/3307311912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 msgr2=0x7fe8800f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 --2- 172.21.15.169:0/3307311912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8800f58a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe868009fb0 tx=0x7fe868009b30).stop 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 -- 172.21.15.169:0/3307311912 shutdown_connections 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 --2- 172.21.15.169:0/3307311912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8800f58a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 -- 172.21.15.169:0/3307311912 >> 172.21.15.169:0/3307311912 conn(0x7fe8800f0be0 msgr2=0x7fe8800f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 -- 172.21.15.169:0/3307311912 shutdown_connections 2024-01-11T05:10:31.213 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.212+0000 7fe885007700 1 -- 172.21.15.169:0/3307311912 wait complete. 2024-01-11T05:10:31.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.213+0000 7fe885007700 1 Processor -- start 2024-01-11T05:10:31.215 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.213+0000 7fe885007700 1 -- start start 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.213+0000 7fe885007700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8801056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.213+0000 7fe885007700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe880105be0 con 0x7fe8800f5480 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe87ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8801056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe87ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8801056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45366/0 (socket says 172.21.15.169:45366) 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe87ed9d700 1 -- 172.21.15.169:0/2861735533 learned_addr learned my addr 172.21.15.169:0/2861735533 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe87ed9d700 1 -- 172.21.15.169:0/2861735533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe868005040 con 0x7fe8800f5480 2024-01-11T05:10:31.216 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe87ed9d700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8801056f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fe8680089f0 tx=0x7fe86800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe868009320 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe868013ec0 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe880105de0 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe868014d30 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.214+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe880106270 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.215+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe868014420 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.215+0000 7fe8777fe700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe86c031090 0x7fe86c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.215+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8680424c0 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.215+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe880045bb0 con 0x7fe8800f5480 2024-01-11T05:10:31.217 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.215+0000 7fe87e59c700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe86c031090 0x7fe86c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:31.218 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.216+0000 7fe87e59c700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe86c031090 0x7fe86c033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe8700096a0 tx=0x7fe870006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:31.220 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.220+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe868014750 con 0x7fe8800f5480 2024-01-11T05:10:31.430 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.429+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8800fb060 con 0x7fe8800f5480 2024-01-11T05:10:31.431 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.430+0000 7fe8777fe700 1 -- 172.21.15.169:0/2861735533 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe86801d380 con 0x7fe8800f5480 2024-01-11T05:10:31.431 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:31.431 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe86c031090 msgr2=0x7fe86c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe86c031090 0x7fe86c033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe8700096a0 tx=0x7fe870006b40).stop 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 msgr2=0x7fe8801056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8801056f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fe8680089f0 tx=0x7fe86800b040).stop 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 shutdown_connections 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fe86c031090 0x7fe86c033550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 --2- 172.21.15.169:0/2861735533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8800f5480 0x7fe8801056f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 >> 172.21.15.169:0/2861735533 conn(0x7fe8800f0be0 msgr2=0x7fe8800f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:31.433 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 shutdown_connections 2024-01-11T05:10:31.434 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:31.432+0000 7fe885007700 1 -- 172.21.15.169:0/2861735533 wait complete. 2024-01-11T05:10:31.434 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:30.338079+0000 mgr.smithi067.srbzpc (mgr.14156) 222 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:32.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:32 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:31.430653+0000 mon.smithi067 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.169:0/2861735533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:32.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:32.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:32.338435+0000 mgr.smithi067.srbzpc (mgr.14156) 223 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:33.881 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 -- 172.21.15.169:0/1040500135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 msgr2=0x7f0c900f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:33.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 --2- 172.21.15.169:0/1040500135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c900f58a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f0c84009fb0 tx=0x7f0c84009b30).stop 2024-01-11T05:10:33.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 -- 172.21.15.169:0/1040500135 shutdown_connections 2024-01-11T05:10:33.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 --2- 172.21.15.169:0/1040500135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c900f58a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:33.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 -- 172.21.15.169:0/1040500135 >> 172.21.15.169:0/1040500135 conn(0x7f0c900f0be0 msgr2=0x7f0c900f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:33.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 -- 172.21.15.169:0/1040500135 shutdown_connections 2024-01-11T05:10:33.882 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.880+0000 7f0c95819700 1 -- 172.21.15.169:0/1040500135 wait complete. 2024-01-11T05:10:33.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c95819700 1 Processor -- start 2024-01-11T05:10:33.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c95819700 1 -- start start 2024-01-11T05:10:33.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c95819700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c901089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:33.883 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c95819700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c90108ed0 con 0x7f0c900f5480 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c8effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c901089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c8effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c901089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45370/0 (socket says 172.21.15.169:45370) 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.881+0000 7f0c8effd700 1 -- 172.21.15.169:0/4278212211 learned_addr learned my addr 172.21.15.169:0/4278212211 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c8effd700 1 -- 172.21.15.169:0/4278212211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84005040 con 0x7f0c900f5480 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c8effd700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c901089e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0c840089f0 tx=0x7f0c8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c84009320 con 0x7f0c900f5480 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c901090d0 con 0x7f0c900f5480 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0c84013ec0 con 0x7f0c900f5480 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c84014d30 con 0x7f0c900f5480 2024-01-11T05:10:33.884 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.882+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c90109560 con 0x7f0c900f5480 2024-01-11T05:10:33.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.883+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0c840263a0 con 0x7f0c900f5480 2024-01-11T05:10:33.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.883+0000 7f0c777fe700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0c78031090 0x7f0c78033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:33.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.883+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0c84042470 con 0x7f0c900f5480 2024-01-11T05:10:33.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.883+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c900591e0 con 0x7f0c900f5480 2024-01-11T05:10:33.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.883+0000 7f0c8e7fc700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0c78031090 0x7f0c78033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:33.885 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.884+0000 7f0c8e7fc700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0c78031090 0x7f0c78033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f0c800096a0 tx=0x7f0c80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:33.889 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:33.888+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0c84012920 con 0x7f0c900f5480 2024-01-11T05:10:34.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.099+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c90045bb0 con 0x7f0c900f5480 2024-01-11T05:10:34.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.100+0000 7f0c777fe700 1 -- 172.21.15.169:0/4278212211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c8402b020 con 0x7f0c900f5480 2024-01-11T05:10:34.101 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:34.101 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.101+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0c78031090 msgr2=0x7f0c78033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.101+0000 7f0c95819700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0c78031090 0x7f0c78033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f0c800096a0 tx=0x7f0c80006b40).stop 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.101+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 msgr2=0x7f0c901089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.101+0000 7f0c95819700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c901089e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0c840089f0 tx=0x7f0c8400b040).stop 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.102+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 shutdown_connections 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.102+0000 7f0c95819700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f0c78031090 0x7f0c78033550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.102+0000 7f0c95819700 1 --2- 172.21.15.169:0/4278212211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c900f5480 0x7f0c901089e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:34.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.102+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 >> 172.21.15.169:0/4278212211 conn(0x7f0c900f0be0 msgr2=0x7f0c9005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:34.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.102+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 shutdown_connections 2024-01-11T05:10:34.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:34.102+0000 7f0c95819700 1 -- 172.21.15.169:0/4278212211 wait complete. 2024-01-11T05:10:34.104 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:35.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:35.455 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:34.100982+0000 mon.smithi067 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.169:0/4278212211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:35.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:34.338806+0000 mgr.smithi067.srbzpc (mgr.14156) 224 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.551+0000 7f418b332700 1 -- 172.21.15.169:0/2512093341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 msgr2=0x7f41840f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.551+0000 7f418b332700 1 --2- 172.21.15.169:0/2512093341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41840f69b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4174004760 tx=0x7f4174009ad0).stop 2024-01-11T05:10:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.551+0000 7f418b332700 1 -- 172.21.15.169:0/2512093341 shutdown_connections 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.551+0000 7f418b332700 1 --2- 172.21.15.169:0/2512093341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41840f69b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.551+0000 7f418b332700 1 -- 172.21.15.169:0/2512093341 >> 172.21.15.169:0/2512093341 conn(0x7f41840f1cd0 msgr2=0x7f41840f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.551+0000 7f418b332700 1 -- 172.21.15.169:0/2512093341 shutdown_connections 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.552+0000 7f418b332700 1 -- 172.21.15.169:0/2512093341 wait complete. 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.552+0000 7f418b332700 1 Processor -- start 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.552+0000 7f418b332700 1 -- start start 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.552+0000 7f418b332700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41841067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.552+0000 7f418b332700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4184106cb0 con 0x7f41840f6590 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f41890ce700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41841067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f41890ce700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41841067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45392/0 (socket says 172.21.15.169:45392) 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f41890ce700 1 -- 172.21.15.169:0/1619737940 learned_addr learned my addr 172.21.15.169:0/1619737940 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f41890ce700 1 -- 172.21.15.169:0/1619737940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4174009d70 con 0x7f41840f6590 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f41890ce700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41841067c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f4174004170 tx=0x7f4174009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417400e040 con 0x7f41840f6590 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4184106eb0 con 0x7f41840f6590 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4174008ef0 con 0x7f41840f6590 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.553+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4174014c90 con 0x7f41840f6590 2024-01-11T05:10:36.556 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.554+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4184107340 con 0x7f41840f6590 2024-01-11T05:10:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.554+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4184045bb0 con 0x7f41840f6590 2024-01-11T05:10:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.555+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4174014420 con 0x7f41840f6590 2024-01-11T05:10:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.555+0000 7f4179ffb700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4170031090 0x7f4170033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.555+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4174042240 con 0x7f41840f6590 2024-01-11T05:10:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.555+0000 7f41888cd700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4170031090 0x7f4170033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:36.557 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.555+0000 7f41888cd700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4170031090 0x7f4170033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f41800096a0 tx=0x7f4180006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:36.561 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.560+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4174026300 con 0x7f41840f6590 2024-01-11T05:10:36.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.770+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41841081f0 con 0x7f41840f6590 2024-01-11T05:10:36.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.770+0000 7f4179ffb700 1 -- 172.21.15.169:0/1619737940 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4174012b20 con 0x7f41840f6590 2024-01-11T05:10:36.772 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:36.772 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4170031090 msgr2=0x7f4170033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4170031090 0x7f4170033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f41800096a0 tx=0x7f4180006b40).stop 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 msgr2=0x7f41841067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41841067c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f4174004170 tx=0x7f4174009ad0).stop 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 shutdown_connections 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4170031090 0x7f4170033550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 --2- 172.21.15.169:0/1619737940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41840f6590 0x7f41841067c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:36.773 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 >> 172.21.15.169:0/1619737940 conn(0x7f41840f1cd0 msgr2=0x7f41840f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:36.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 shutdown_connections 2024-01-11T05:10:36.774 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:36.772+0000 7f418b332700 1 -- 172.21.15.169:0/1619737940 wait complete. 2024-01-11T05:10:36.774 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:36.339040+0000 mgr.smithi067.srbzpc (mgr.14156) 225 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:36.771692+0000 mon.smithi067 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.169:0/1619737940' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:38.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:38.147 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:39.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.264+0000 7f8c48466700 1 -- 172.21.15.169:0/4012858377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 msgr2=0x7f8c400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.264+0000 7f8c48466700 1 --2- 172.21.15.169:0/4012858377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c400f58a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f8c34009fb0 tx=0x7f8c34009b30).stop 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.264+0000 7f8c48466700 1 -- 172.21.15.169:0/4012858377 shutdown_connections 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.264+0000 7f8c48466700 1 --2- 172.21.15.169:0/4012858377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c400f58a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.264+0000 7f8c48466700 1 -- 172.21.15.169:0/4012858377 >> 172.21.15.169:0/4012858377 conn(0x7f8c400f0be0 msgr2=0x7f8c400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.264+0000 7f8c48466700 1 -- 172.21.15.169:0/4012858377 shutdown_connections 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c48466700 1 -- 172.21.15.169:0/4012858377 wait complete. 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c48466700 1 Processor -- start 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c48466700 1 -- start start 2024-01-11T05:10:39.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c48466700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c401056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c48466700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c40105be0 con 0x7f8c400f5480 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c46202700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c401056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c46202700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c401056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45402/0 (socket says 172.21.15.169:45402) 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.265+0000 7f8c46202700 1 -- 172.21.15.169:0/1688862661 learned_addr learned my addr 172.21.15.169:0/1688862661 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c46202700 1 -- 172.21.15.169:0/1688862661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c34005040 con 0x7f8c400f5480 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c46202700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c401056f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f8c340089f0 tx=0x7f8c3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c34009320 con 0x7f8c400f5480 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8c34013ec0 con 0x7f8c400f5480 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c40105de0 con 0x7f8c400f5480 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c34014d30 con 0x7f8c400f5480 2024-01-11T05:10:39.269 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.266+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c40106270 con 0x7f8c400f5480 2024-01-11T05:10:39.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.267+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8c340263a0 con 0x7f8c400f5480 2024-01-11T05:10:39.270 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.267+0000 7f8c32ffd700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c2c0310e0 0x7f8c2c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:39.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.267+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c40045bb0 con 0x7f8c400f5480 2024-01-11T05:10:39.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.267+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8c34042360 con 0x7f8c400f5480 2024-01-11T05:10:39.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.267+0000 7f8c45a01700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c2c0310e0 0x7f8c2c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:39.271 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.268+0000 7f8c45a01700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c2c0310e0 0x7f8c2c0335a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8c3c0096a0 tx=0x7f8c3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:39.273 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.272+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8c3401cb10 con 0x7f8c400f5480 2024-01-11T05:10:39.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.478+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c400fb060 con 0x7f8c400f5480 2024-01-11T05:10:39.480 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.479+0000 7f8c32ffd700 1 -- 172.21.15.169:0/1688862661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c34034020 con 0x7f8c400f5480 2024-01-11T05:10:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:39.480 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:39.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c2c0310e0 msgr2=0x7f8c2c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c2c0310e0 0x7f8c2c0335a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8c3c0096a0 tx=0x7f8c3c006b40).stop 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 msgr2=0x7f8c401056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c401056f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f8c340089f0 tx=0x7f8c3400b040).stop 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 shutdown_connections 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c2c0310e0 0x7f8c2c0335a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 --2- 172.21.15.169:0/1688862661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c400f5480 0x7f8c401056f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 >> 172.21.15.169:0/1688862661 conn(0x7f8c400f0be0 msgr2=0x7f8c400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 shutdown_connections 2024-01-11T05:10:39.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:39.480+0000 7f8c48466700 1 -- 172.21.15.169:0/1688862661 wait complete. 2024-01-11T05:10:39.483 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:39.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:38.339453+0000 mgr.smithi067.srbzpc (mgr.14156) 226 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:40.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:40 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:39.479951+0000 mon.smithi067 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.169:0/1688862661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:40.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:40.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:40.339869+0000 mgr.smithi067.srbzpc (mgr.14156) 227 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:41.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 -- 172.21.15.169:0/3150017294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 msgr2=0x7f45500f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 --2- 172.21.15.169:0/3150017294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45500f58d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f4540009fb0 tx=0x7f4540009b30).stop 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 -- 172.21.15.169:0/3150017294 shutdown_connections 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 --2- 172.21.15.169:0/3150017294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45500f58d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 -- 172.21.15.169:0/3150017294 >> 172.21.15.169:0/3150017294 conn(0x7f45500f0c10 msgr2=0x7f45500f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 -- 172.21.15.169:0/3150017294 shutdown_connections 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.942+0000 7f45569e2700 1 -- 172.21.15.169:0/3150017294 wait complete. 2024-01-11T05:10:41.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f45569e2700 1 Processor -- start 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f45569e2700 1 -- start start 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f45569e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45501094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f45569e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550060360 con 0x7f45500f54b0 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f454ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45501094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f454ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45501094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46514/0 (socket says 172.21.15.169:46514) 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.943+0000 7f454ffff700 1 -- 172.21.15.169:0/613177115 learned_addr learned my addr 172.21.15.169:0/613177115 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f454ffff700 1 -- 172.21.15.169:0/613177115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540005040 con 0x7f45500f54b0 2024-01-11T05:10:41.947 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f454ffff700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45501094c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f45400089f0 tx=0x7f454000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:41.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540009320 con 0x7f45500f54b0 2024-01-11T05:10:41.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4550060560 con 0x7f45500f54b0 2024-01-11T05:10:41.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4540013ec0 con 0x7f45500f54b0 2024-01-11T05:10:41.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540014d30 con 0x7f45500f54b0 2024-01-11T05:10:41.948 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.944+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4550109bf0 con 0x7f45500f54b0 2024-01-11T05:10:41.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.945+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4540014420 con 0x7f45500f54b0 2024-01-11T05:10:41.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.945+0000 7f454cff9700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4538031090 0x7f4538033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:41.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.945+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4550045bb0 con 0x7f45500f54b0 2024-01-11T05:10:41.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.945+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f45400424b0 con 0x7f45500f54b0 2024-01-11T05:10:41.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.945+0000 7f454f7fe700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4538031090 0x7f4538033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:41.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.946+0000 7f454f7fe700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4538031090 0x7f4538033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f45440096a0 tx=0x7f4544006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:41.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:41.950+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4540014750 con 0x7f45500f54b0 2024-01-11T05:10:42.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.158+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f455010a440 con 0x7f45500f54b0 2024-01-11T05:10:42.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.158+0000 7f454cff9700 1 -- 172.21.15.169:0/613177115 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f454000fe40 con 0x7f45500f54b0 2024-01-11T05:10:42.160 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:42.160 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:42.161 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4538031090 msgr2=0x7f4538033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4538031090 0x7f4538033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f45440096a0 tx=0x7f4544006b40).stop 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 msgr2=0x7f45501094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45501094c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f45400089f0 tx=0x7f454000b040).stop 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 shutdown_connections 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4538031090 0x7f4538033550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 --2- 172.21.15.169:0/613177115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45500f54b0 0x7f45501094c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 >> 172.21.15.169:0/613177115 conn(0x7f45500f0c10 msgr2=0x7f45500f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 shutdown_connections 2024-01-11T05:10:42.162 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:42.160+0000 7f45569e2700 1 -- 172.21.15.169:0/613177115 wait complete. 2024-01-11T05:10:42.163 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:42.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:42 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:42.159918+0000 mon.smithi067 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.169:0/613177115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:43.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:43.527 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:43.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:42.340198+0000 mgr.smithi067.srbzpc (mgr.14156) 228 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 -- 172.21.15.169:0/3752127616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 msgr2=0x7f80d80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 --2- 172.21.15.169:0/3752127616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d80f58a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f80c0009fb0 tx=0x7f80c0009b30).stop 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 -- 172.21.15.169:0/3752127616 shutdown_connections 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 --2- 172.21.15.169:0/3752127616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d80f58a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 -- 172.21.15.169:0/3752127616 >> 172.21.15.169:0/3752127616 conn(0x7f80d80f0be0 msgr2=0x7f80d80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 -- 172.21.15.169:0/3752127616 shutdown_connections 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.643+0000 7f80dcc52700 1 -- 172.21.15.169:0/3752127616 wait complete. 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80dcc52700 1 Processor -- start 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80dcc52700 1 -- start start 2024-01-11T05:10:44.646 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80dcc52700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d81094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80dcc52700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d8060320 con 0x7f80d80f5480 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80d659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d81094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80d659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d81094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46536/0 (socket says 172.21.15.169:46536) 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80d659c700 1 -- 172.21.15.169:0/1917495387 learned_addr learned my addr 172.21.15.169:0/1917495387 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80d659c700 1 -- 172.21.15.169:0/1917495387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0005040 con 0x7f80d80f5480 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.644+0000 7f80d659c700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d81094f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f80c0000c00 tx=0x7f80c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.645+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80c00095f0 con 0x7f80d80f5480 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.645+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d8060520 con 0x7f80d80f5480 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.645+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f80c0007e50 con 0x7f80d80f5480 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.645+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80c0014e70 con 0x7f80d80f5480 2024-01-11T05:10:44.647 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.645+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d8109ba0 con 0x7f80d80f5480 2024-01-11T05:10:44.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.645+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f80c0025070 con 0x7f80d80f5480 2024-01-11T05:10:44.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.646+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d8045bb0 con 0x7f80d80f5480 2024-01-11T05:10:44.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.646+0000 7f80ceffd700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80c4031090 0x7f80c4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:44.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.646+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f80c0041e00 con 0x7f80d80f5480 2024-01-11T05:10:44.648 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.646+0000 7f80d5d9b700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80c4031090 0x7f80c4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:44.649 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.646+0000 7f80d5d9b700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80c4031090 0x7f80c4033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f80c80096a0 tx=0x7f80c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:44.652 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.651+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f80c000f300 con 0x7f80d80f5480 2024-01-11T05:10:44.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.858+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80d80faf70 con 0x7f80d80f5480 2024-01-11T05:10:44.860 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.859+0000 7f80ceffd700 1 -- 172.21.15.169:0/1917495387 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80c000f300 con 0x7f80d80f5480 2024-01-11T05:10:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:44.861 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:44.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.860+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80c4031090 msgr2=0x7f80c4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:44.862 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.860+0000 7f80dcc52700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80c4031090 0x7f80c4033550 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f80c80096a0 tx=0x7f80c8006b40).stop 2024-01-11T05:10:44.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.860+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 msgr2=0x7f80d81094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:44.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.860+0000 7f80dcc52700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d81094f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f80c0000c00 tx=0x7f80c000b040).stop 2024-01-11T05:10:44.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.860+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 shutdown_connections 2024-01-11T05:10:44.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.861+0000 7f80dcc52700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80c4031090 0x7f80c4033550 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:44.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.861+0000 7f80dcc52700 1 --2- 172.21.15.169:0/1917495387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d80f5480 0x7f80d81094f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:44.863 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.861+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 >> 172.21.15.169:0/1917495387 conn(0x7f80d80f0be0 msgr2=0x7f80d80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:44.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.861+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 shutdown_connections 2024-01-11T05:10:44.864 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:44.861+0000 7f80dcc52700 1 -- 172.21.15.169:0/1917495387 wait complete. 2024-01-11T05:10:44.864 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:44.340498+0000 mgr.smithi067.srbzpc (mgr.14156) 229 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:44.860348+0000 mon.smithi067 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.169:0/1917495387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:46.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:46.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:47.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 -- 172.21.15.169:0/287704310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 msgr2=0x7fdbc40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:47.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 --2- 172.21.15.169:0/287704310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc40f69b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdbb4004760 tx=0x7fdbb4009ad0).stop 2024-01-11T05:10:47.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 -- 172.21.15.169:0/287704310 shutdown_connections 2024-01-11T05:10:47.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 --2- 172.21.15.169:0/287704310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc40f69b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 -- 172.21.15.169:0/287704310 >> 172.21.15.169:0/287704310 conn(0x7fdbc40f1cd0 msgr2=0x7fdbc40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 -- 172.21.15.169:0/287704310 shutdown_connections 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.352+0000 7fdbcb498700 1 -- 172.21.15.169:0/287704310 wait complete. 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbcb498700 1 Processor -- start 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbcb498700 1 -- start start 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbcb498700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbcb498700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc4106cb0 con 0x7fdbc40f6590 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbc9234700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbc9234700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:46564/0 (socket says 172.21.15.169:46564) 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.353+0000 7fdbc9234700 1 -- 172.21.15.169:0/640801607 learned_addr learned my addr 172.21.15.169:0/640801607 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbc9234700 1 -- 172.21.15.169:0/640801607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4009d70 con 0x7fdbc40f6590 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbc9234700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc41067c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fdbb4004170 tx=0x7fdbb4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:47.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb400e040 con 0x7fdbc40f6590 2024-01-11T05:10:47.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc4106eb0 con 0x7fdbc40f6590 2024-01-11T05:10:47.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdbb4008ef0 con 0x7fdbc40f6590 2024-01-11T05:10:47.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb4014c90 con 0x7fdbc40f6590 2024-01-11T05:10:47.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.354+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc4107340 con 0x7fdbc40f6590 2024-01-11T05:10:47.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.355+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdbb4014420 con 0x7fdbc40f6590 2024-01-11T05:10:47.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.355+0000 7fdbb9ffb700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdbb00310e0 0x7fdbb00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:47.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.355+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc4045bb0 con 0x7fdbc40f6590 2024-01-11T05:10:47.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.355+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdbb4042320 con 0x7fdbc40f6590 2024-01-11T05:10:47.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.355+0000 7fdbc8a33700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdbb00310e0 0x7fdbb00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:47.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.356+0000 7fdbc8a33700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdbb00310e0 0x7fdbb00335a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdbc00096a0 tx=0x7fdbc0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:47.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.360+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdbb4026300 con 0x7fdbc40f6590 2024-01-11T05:10:47.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.570+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbc4108190 con 0x7fdbc40f6590 2024-01-11T05:10:47.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.570+0000 7fdbb9ffb700 1 -- 172.21.15.169:0/640801607 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbb4012b20 con 0x7fdbc40f6590 2024-01-11T05:10:47.571 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:47.572 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdbb00310e0 msgr2=0x7fdbb00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdbb00310e0 0x7fdbb00335a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdbc00096a0 tx=0x7fdbc0006b40).stop 2024-01-11T05:10:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 msgr2=0x7fdbc41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc41067c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fdbb4004170 tx=0x7fdbb4009ad0).stop 2024-01-11T05:10:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 shutdown_connections 2024-01-11T05:10:47.573 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdbb00310e0 0x7fdbb00335a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 --2- 172.21.15.169:0/640801607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc40f6590 0x7fdbc41067c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 >> 172.21.15.169:0/640801607 conn(0x7fdbc40f1cd0 msgr2=0x7fdbc40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 shutdown_connections 2024-01-11T05:10:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:47.572+0000 7fdbcb498700 1 -- 172.21.15.169:0/640801607 wait complete. 2024-01-11T05:10:47.574 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:46.340862+0000 mgr.smithi067.srbzpc (mgr.14156) 230 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:48.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:48 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:47.571551+0000 mon.smithi067 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.169:0/640801607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:48.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:48.962 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:48.341158+0000 mgr.smithi067.srbzpc (mgr.14156) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:50.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1928520651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 msgr2=0x7f43b00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 --2- 172.21.15.169:0/1928520651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b00f69b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f4398004760 tx=0x7f4398009ad0).stop 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1928520651 shutdown_connections 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 --2- 172.21.15.169:0/1928520651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b00f69b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1928520651 >> 172.21.15.169:0/1928520651 conn(0x7f43b00f1cd0 msgr2=0x7f43b00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1928520651 shutdown_connections 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1928520651 wait complete. 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 Processor -- start 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- start start 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.061+0000 7f43b4c8a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0106cb0 con 0x7f43b00f6590 2024-01-11T05:10:50.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:50.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:41306/0 (socket says 172.21.15.169:41306) 2024-01-11T05:10:50.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43ae59c700 1 -- 172.21.15.169:0/1126463397 learned_addr learned my addr 172.21.15.169:0/1126463397 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:50.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43ae59c700 1 -- 172.21.15.169:0/1126463397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4398009d70 con 0x7f43b00f6590 2024-01-11T05:10:50.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43ae59c700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b01067c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f4398004170 tx=0x7f4398009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:50.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f439800e040 con 0x7f43b00f6590 2024-01-11T05:10:50.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b0106eb0 con 0x7f43b00f6590 2024-01-11T05:10:50.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4398008ef0 con 0x7f43b00f6590 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4398014c90 con 0x7f43b00f6590 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.062+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b0107340 con 0x7f43b00f6590 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.063+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4398014420 con 0x7f43b00f6590 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.063+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b0045bb0 con 0x7f43b00f6590 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.063+0000 7f43a6ffd700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f439c0310e0 0x7f439c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.064+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4398042300 con 0x7f43b00f6590 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.064+0000 7f43add9b700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f439c0310e0 0x7f439c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.064+0000 7f43add9b700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f439c0310e0 0x7f439c0335a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f43a00096a0 tx=0x7f43a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:50.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.069+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4398012d00 con 0x7f43b00f6590 2024-01-11T05:10:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.280+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43b01081f0 con 0x7f43b00f6590 2024-01-11T05:10:50.281 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.280+0000 7f43a6ffd700 1 -- 172.21.15.169:0/1126463397 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4398012b20 con 0x7f43b00f6590 2024-01-11T05:10:50.281 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:50.282 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f439c0310e0 msgr2=0x7f439c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f439c0310e0 0x7f439c0335a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f43a00096a0 tx=0x7f43a0006b40).stop 2024-01-11T05:10:50.283 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 msgr2=0x7f43b01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b01067c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f4398004170 tx=0x7f4398009ad0).stop 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 shutdown_connections 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f439c0310e0 0x7f439c0335a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 --2- 172.21.15.169:0/1126463397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43b00f6590 0x7f43b01067c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 >> 172.21.15.169:0/1126463397 conn(0x7f43b00f1cd0 msgr2=0x7f43b00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 shutdown_connections 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:50.282+0000 7f43b4c8a700 1 -- 172.21.15.169:0/1126463397 wait complete. 2024-01-11T05:10:50.284 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:50.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:50.281585+0000 mon.smithi067 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.169:0/1126463397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:51.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:51.649 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:51.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:51 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:50.341508+0000 mgr.smithi067.srbzpc (mgr.14156) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 -- 172.21.15.169:0/1320800464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 msgr2=0x7f74d40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 --2- 172.21.15.169:0/1320800464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d40f58a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f74c8009fb0 tx=0x7f74c8009b30).stop 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 -- 172.21.15.169:0/1320800464 shutdown_connections 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 --2- 172.21.15.169:0/1320800464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d40f58a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 -- 172.21.15.169:0/1320800464 >> 172.21.15.169:0/1320800464 conn(0x7f74d40f0be0 msgr2=0x7f74d40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 -- 172.21.15.169:0/1320800464 shutdown_connections 2024-01-11T05:10:52.768 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 -- 172.21.15.169:0/1320800464 wait complete. 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 Processor -- start 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 -- start start 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.767+0000 7f74d9057700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d41056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d9057700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d4105be0 con 0x7f74d40f5480 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d41056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d41056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:41328/0 (socket says 172.21.15.169:41328) 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d2d9d700 1 -- 172.21.15.169:0/2959324337 learned_addr learned my addr 172.21.15.169:0/2959324337 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d2d9d700 1 -- 172.21.15.169:0/2959324337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c8005040 con 0x7f74d40f5480 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d2d9d700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d41056f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f74c80089f0 tx=0x7f74c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c8009320 con 0x7f74d40f5480 2024-01-11T05:10:52.770 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d4105de0 con 0x7f74d40f5480 2024-01-11T05:10:52.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f74c8013ec0 con 0x7f74d40f5480 2024-01-11T05:10:52.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c8014d30 con 0x7f74d40f5480 2024-01-11T05:10:52.771 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.768+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d4106270 con 0x7f74d40f5480 2024-01-11T05:10:52.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.769+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f74c80263a0 con 0x7f74d40f5480 2024-01-11T05:10:52.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.769+0000 7f74bb7fe700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74bc031090 0x7f74bc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:52.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.769+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f74c8042410 con 0x7f74d40f5480 2024-01-11T05:10:52.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.769+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d4045bb0 con 0x7f74d40f5480 2024-01-11T05:10:52.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.770+0000 7f74d259c700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74bc031090 0x7f74bc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:52.772 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.770+0000 7f74d259c700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74bc031090 0x7f74bc033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f74c40096a0 tx=0x7f74c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:52.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.774+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f74c80266d0 con 0x7f74d40f5480 2024-01-11T05:10:52.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.982+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74d40fb060 con 0x7f74d40f5480 2024-01-11T05:10:52.983 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.982+0000 7f74bb7fe700 1 -- 172.21.15.169:0/2959324337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74c802b020 con 0x7f74d40f5480 2024-01-11T05:10:52.984 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:52.984 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:52.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74bc031090 msgr2=0x7f74bc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:52.985 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74bc031090 0x7f74bc033550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f74c40096a0 tx=0x7f74c4006b40).stop 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 msgr2=0x7f74d41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d41056f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f74c80089f0 tx=0x7f74c800b040).stop 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 shutdown_connections 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f74bc031090 0x7f74bc033550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 --2- 172.21.15.169:0/2959324337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d40f5480 0x7f74d41056f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 >> 172.21.15.169:0/2959324337 conn(0x7f74d40f0be0 msgr2=0x7f74d40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 shutdown_connections 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:52.984+0000 7f74d9057700 1 -- 172.21.15.169:0/2959324337 wait complete. 2024-01-11T05:10:52.986 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:53.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:52.341937+0000 mgr.smithi067.srbzpc (mgr.14156) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:53.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:10:52.553063+0000 mgr.smithi067.srbzpc (mgr.14156) 234 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-89773ea0-b03e-11ee-95ab-87774f69a715-node-exporter.smithi067 2024-01-11T05:10:53.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: /bin/podman: stderr Error: no such container ceph-89773ea0-b03e-11ee-95ab-87774f69a715-node-exporter.smithi067 2024-01-11T05:10:53.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Deploy daemon node-exporter.smithi067 ... 2024-01-11T05:10:53.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Verifying port 9100 ... 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 1 from systemctl start ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: systemctl: stderr Job for ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067.service failed because a timeout was exceeded. 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: systemctl: stderr See "systemctl status ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067.service" and "journalctl -xe" for details. 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: main() 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: r = ctx.func(ctx) 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: return func(ctx) 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ports=daemon_ports) 2024-01-11T05:10:53.764 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: RuntimeError: Failed command: systemctl start ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: yield (conn, connr) 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: code, '\n'.join(err))) 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-89773ea0-b03e-11ee-95ab-87774f69a715-node-exporter.smithi067 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: /bin/podman: stderr Error: no such container ceph-89773ea0-b03e-11ee-95ab-87774f69a715-node-exporter.smithi067 2024-01-11T05:10:53.765 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Deploy daemon node-exporter.smithi067 ... 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Verifying port 9100 ... 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 1 from systemctl start ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: systemctl: stderr Job for ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067.service failed because a timeout was exceeded. 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: systemctl: stderr See "systemctl status ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067.service" and "journalctl -xe" for details. 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: main() 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: r = ctx.func(ctx) 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: return func(ctx) 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ports=daemon_ports) 2024-01-11T05:10:53.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: RuntimeError: Failed command: systemctl start ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:10:52.553271+0000 mgr.smithi067.srbzpc (mgr.14156) 235 : cephadm [ERR] Failed while placing node-exporter.smithi067 on smithi067: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-89773ea0-b03e-11ee-95ab-87774f69a715-node-exporter.smithi067 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: /bin/podman: stderr Error: no such container ceph-89773ea0-b03e-11ee-95ab-87774f69a715-node-exporter.smithi067 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Deploy daemon node-exporter.smithi067 ... 2024-01-11T05:10:53.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Verifying port 9100 ... 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Non-zero exit code 1 from systemctl start ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: systemctl: stderr Job for ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067.service failed because a timeout was exceeded. 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: systemctl: stderr See "systemctl status ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067.service" and "journalctl -xe" for details. 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: Traceback (most recent call last): 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: main() 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: r = ctx.func(ctx) 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: return func(ctx) 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: ports=daemon_ports) 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2024-01-11T05:10:53.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: File "/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: RuntimeError: Failed command: systemctl start ceph-89773ea0-b03e-11ee-95ab-87774f69a715@node-exporter.smithi067 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cephadm 2024-01-11T05:10:52.559346+0000 mgr.smithi067.srbzpc (mgr.14156) 236 : cephadm [INF] Deploying daemon prometheus.smithi067 on smithi067 2024-01-11T05:10:53.769 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:53 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:52.983918+0000 mon.smithi067 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.169:0/2959324337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:54.368 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:54.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.477+0000 7f70443c7700 1 -- 172.21.15.169:0/2159373358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 msgr2=0x7f703c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:55.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.477+0000 7f70443c7700 1 --2- 172.21.15.169:0/2159373358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c0f69b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f702c009fb0 tx=0x7f702c009b30).stop 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.477+0000 7f70443c7700 1 -- 172.21.15.169:0/2159373358 shutdown_connections 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.477+0000 7f70443c7700 1 --2- 172.21.15.169:0/2159373358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c0f69b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.477+0000 7f70443c7700 1 -- 172.21.15.169:0/2159373358 >> 172.21.15.169:0/2159373358 conn(0x7f703c0f1d10 msgr2=0x7f703c0f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f70443c7700 1 -- 172.21.15.169:0/2159373358 shutdown_connections 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f70443c7700 1 -- 172.21.15.169:0/2159373358 wait complete. 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f70443c7700 1 Processor -- start 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f70443c7700 1 -- start start 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f70443c7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f70443c7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c106cf0 con 0x7f703c0f6590 2024-01-11T05:10:55.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f7042163700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f7042163700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:41346/0 (socket says 172.21.15.169:41346) 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.478+0000 7f7042163700 1 -- 172.21.15.169:0/333776310 learned_addr learned my addr 172.21.15.169:0/333776310 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f7042163700 1 -- 172.21.15.169:0/333776310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c005040 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f7042163700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c106800 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f702c0089f0 tx=0x7f702c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f702c009320 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c106ef0 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f702c013ec0 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f702c014d30 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.479+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c107380 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.480+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f702c0095f0 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.480+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c045bb0 con 0x7f703c0f6590 2024-01-11T05:10:55.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.480+0000 7f7032ffd700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7028031090 0x7f7028033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:55.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.480+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f702c0423e0 con 0x7f703c0f6590 2024-01-11T05:10:55.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.480+0000 7f7041962700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7028031090 0x7f7028033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:55.484 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.481+0000 7f7041962700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7028031090 0x7f7028033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f70380096a0 tx=0x7f7038006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:55.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.485+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f702c01d980 con 0x7f703c0f6590 2024-01-11T05:10:55.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:55 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:54.342396+0000 mgr.smithi067.srbzpc (mgr.14156) 237 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:55.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.691+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f703c0fc080 con 0x7f703c0f6590 2024-01-11T05:10:55.693 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.692+0000 7f7032ffd700 1 -- 172.21.15.169:0/333776310 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f702c01d380 con 0x7f703c0f6590 2024-01-11T05:10:55.693 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:55.694 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:55.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7028031090 msgr2=0x7f7028033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:55.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7028031090 0x7f7028033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f70380096a0 tx=0x7f7038006b40).stop 2024-01-11T05:10:55.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 msgr2=0x7f703c106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:55.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c106800 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f702c0089f0 tx=0x7f702c00b040).stop 2024-01-11T05:10:55.695 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 shutdown_connections 2024-01-11T05:10:55.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f7028031090 0x7f7028033550 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:55.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 --2- 172.21.15.169:0/333776310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f703c0f6590 0x7f703c106800 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:55.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 >> 172.21.15.169:0/333776310 conn(0x7f703c0f1d10 msgr2=0x7f703c0f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:55.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 shutdown_connections 2024-01-11T05:10:55.696 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:55.694+0000 7f70443c7700 1 -- 172.21.15.169:0/333776310 wait complete. 2024-01-11T05:10:55.696 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:56.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:56 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:55.693387+0000 mon.smithi067 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.169:0/333776310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:57.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:57.064 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:10:57.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:57 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:56.342847+0000 mgr.smithi067.srbzpc (mgr.14156) 238 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:58.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 -- 172.21.15.169:0/1117492743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 msgr2=0x7fd0940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:58.188 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 --2- 172.21.15.169:0/1117492743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940f58a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fd084009fb0 tx=0x7fd084009b30).stop 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 -- 172.21.15.169:0/1117492743 shutdown_connections 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 --2- 172.21.15.169:0/1117492743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940f58a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 -- 172.21.15.169:0/1117492743 >> 172.21.15.169:0/1117492743 conn(0x7fd0940f0be0 msgr2=0x7fd0940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 -- 172.21.15.169:0/1117492743 shutdown_connections 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.185+0000 7fd09b725700 1 -- 172.21.15.169:0/1117492743 wait complete. 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd09b725700 1 Processor -- start 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd09b725700 1 -- start start 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd09b725700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd09b725700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940fb8e0 con 0x7fd0940f5480 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd0994c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd0994c1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:41372/0 (socket says 172.21.15.169:41372) 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd0994c1700 1 -- 172.21.15.169:0/3465857837 learned_addr learned my addr 172.21.15.169:0/3465857837 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd0994c1700 1 -- 172.21.15.169:0/3465857837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd084005040 con 0x7fd0940f5480 2024-01-11T05:10:58.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.186+0000 7fd0994c1700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940fd460 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd084000c00 tx=0x7fd08400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:58.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.187+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0840095f0 con 0x7fd0940f5480 2024-01-11T05:10:58.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.187+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd084007e50 con 0x7fd0940f5480 2024-01-11T05:10:58.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.187+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0940fba80 con 0x7fd0940f5480 2024-01-11T05:10:58.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.187+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd084014e70 con 0x7fd0940f5480 2024-01-11T05:10:58.190 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.187+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0940fbf10 con 0x7fd0940f5480 2024-01-11T05:10:58.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.188+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd084025070 con 0x7fd0940f5480 2024-01-11T05:10:58.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.188+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0940591e0 con 0x7fd0940f5480 2024-01-11T05:10:58.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.188+0000 7fd089ffb700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd080031090 0x7fd080033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:10:58.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.188+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd084041dd0 con 0x7fd0940f5480 2024-01-11T05:10:58.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.188+0000 7fd098cc0700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd080031090 0x7fd080033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:10:58.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.189+0000 7fd098cc0700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd080031090 0x7fd080033550 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd0900096a0 tx=0x7fd090006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:10:58.194 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.193+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd08400f4e0 con 0x7fd0940f5480 2024-01-11T05:10:58.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.401+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd094045bb0 con 0x7fd0940f5480 2024-01-11T05:10:58.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.401+0000 7fd089ffb700 1 -- 172.21.15.169:0/3465857837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd08400f300 con 0x7fd0940f5480 2024-01-11T05:10:58.402 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:10:58.403 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:10:58.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd080031090 msgr2=0x7fd080033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:58.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd080031090 0x7fd080033550 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd0900096a0 tx=0x7fd090006b40).stop 2024-01-11T05:10:58.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 msgr2=0x7fd0940fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:10:58.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940fd460 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd084000c00 tx=0x7fd08400b040).stop 2024-01-11T05:10:58.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 shutdown_connections 2024-01-11T05:10:58.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd080031090 0x7fd080033550 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:58.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 --2- 172.21.15.169:0/3465857837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0940f5480 0x7fd0940fd460 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:10:58.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 >> 172.21.15.169:0/3465857837 conn(0x7fd0940f0be0 msgr2=0x7fd09405fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:10:58.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 shutdown_connections 2024-01-11T05:10:58.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:10:58.403+0000 7fd09b725700 1 -- 172.21.15.169:0/3465857837 wait complete. 2024-01-11T05:10:58.405 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:10:58.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:58 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:57.422613+0000 mon.smithi067 (mon.0) 291 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' 2024-01-11T05:10:58.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:58 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:10:58.402715+0000 mon.smithi067 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.169:0/3465857837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:10:59.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:10:59 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:10:58.343195+0000 mgr.smithi067.srbzpc (mgr.14156) 239 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:10:59.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:10:59.779 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 -- 172.21.15.169:0/2636092656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 msgr2=0x7f9ecc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 --2- 172.21.15.169:0/2636092656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc0f69b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f9ebc004760 tx=0x7f9ebc009ad0).stop 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 -- 172.21.15.169:0/2636092656 shutdown_connections 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 --2- 172.21.15.169:0/2636092656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc0f69b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 -- 172.21.15.169:0/2636092656 >> 172.21.15.169:0/2636092656 conn(0x7f9ecc0f1cd0 msgr2=0x7f9ecc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 -- 172.21.15.169:0/2636092656 shutdown_connections 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 -- 172.21.15.169:0/2636092656 wait complete. 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 Processor -- start 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 -- start start 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.891+0000 7f9ed4163700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed4163700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc106cb0 con 0x7f9ecc0f6590 2024-01-11T05:11:00.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed1eff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed1eff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:57190/0 (socket says 172.21.15.169:57190) 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed1eff700 1 -- 172.21.15.169:0/2504420238 learned_addr learned my addr 172.21.15.169:0/2504420238 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed1eff700 1 -- 172.21.15.169:0/2504420238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc009d70 con 0x7f9ecc0f6590 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed1eff700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc1067c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f9ebc013f10 tx=0x7f9ebc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc00e040 con 0x7f9ecc0f6590 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc106eb0 con 0x7f9ecc0f6590 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9ebc007df0 con 0x7f9ecc0f6590 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc014d60 con 0x7f9ecc0f6590 2024-01-11T05:11:00.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.892+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc107340 con 0x7f9ecc0f6590 2024-01-11T05:11:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.893+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9ebc014520 con 0x7f9ecc0f6590 2024-01-11T05:11:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.893+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc045bb0 con 0x7f9ecc0f6590 2024-01-11T05:11:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.893+0000 7f9ec27fc700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eb8031090 0x7f9eb8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.893+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9ebc042580 con 0x7f9ecc0f6590 2024-01-11T05:11:00.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.894+0000 7f9ed16fe700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eb8031090 0x7f9eb8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:00.897 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.894+0000 7f9ed16fe700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eb8031090 0x7f9eb8033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9ec80096a0 tx=0x7f9ec8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:00.899 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:00.898+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9ebc007960 con 0x7f9ecc0f6590 2024-01-11T05:11:01.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.105+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ecc108190 con 0x7f9ecc0f6590 2024-01-11T05:11:01.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.106+0000 7f9ec27fc700 1 -- 172.21.15.169:0/2504420238 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ebc0262f0 con 0x7f9ecc0f6590 2024-01-11T05:11:01.108 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:01.108 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eb8031090 msgr2=0x7f9eb8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eb8031090 0x7f9eb8033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9ec80096a0 tx=0x7f9ec8006b40).stop 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 msgr2=0x7f9ecc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc1067c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f9ebc013f10 tx=0x7f9ebc009ad0).stop 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 shutdown_connections 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f9eb8031090 0x7f9eb8033550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 --2- 172.21.15.169:0/2504420238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ecc0f6590 0x7f9ecc1067c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 >> 172.21.15.169:0/2504420238 conn(0x7f9ecc0f1cd0 msgr2=0x7f9ecc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:01.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 shutdown_connections 2024-01-11T05:11:01.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:01.108+0000 7f9ed4163700 1 -- 172.21.15.169:0/2504420238 wait complete. 2024-01-11T05:11:01.110 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:00.343522+0000 mgr.smithi067.srbzpc (mgr.14156) 240 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:01.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:01 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:01.107466+0000 mon.smithi067 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.169:0/2504420238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:02.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:02.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:03.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 -- 172.21.15.169:0/743050504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 msgr2=0x7f1fc00f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:03.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 --2- 172.21.15.169:0/743050504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00f58d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1fb4009fb0 tx=0x7f1fb4009b30).stop 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 -- 172.21.15.169:0/743050504 shutdown_connections 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 --2- 172.21.15.169:0/743050504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00f58d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 -- 172.21.15.169:0/743050504 >> 172.21.15.169:0/743050504 conn(0x7f1fc00f0c10 msgr2=0x7f1fc00f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 -- 172.21.15.169:0/743050504 shutdown_connections 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.602+0000 7f1fc5aee700 1 -- 172.21.15.169:0/743050504 wait complete. 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.603+0000 7f1fc5aee700 1 Processor -- start 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.603+0000 7f1fc5aee700 1 -- start start 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.603+0000 7f1fc5aee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00fd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:03.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.603+0000 7f1fc5aee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc00fb910 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fbf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fbf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00fd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:57204/0 (socket says 172.21.15.169:57204) 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fbf7fe700 1 -- 172.21.15.169:0/1511990012 learned_addr learned my addr 172.21.15.169:0/1511990012 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fbf7fe700 1 -- 172.21.15.169:0/1511990012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4005040 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fbf7fe700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00fd490 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f1fb4004db0 tx=0x7f1fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4013380 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1fb4007e50 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb4014c30 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc00fbab0 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.604+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc00fbf40 con 0x7f1fc00f54b0 2024-01-11T05:11:03.607 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.605+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1fb4025070 con 0x7f1fc00f54b0 2024-01-11T05:11:03.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.605+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc00591e0 con 0x7f1fc00f54b0 2024-01-11T05:11:03.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.605+0000 7f1fa7fff700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1fa80310e0 0x7f1fa80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:03.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.605+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1fb4041d20 con 0x7f1fc00f54b0 2024-01-11T05:11:03.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.606+0000 7f1fbeffd700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1fa80310e0 0x7f1fa80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:03.608 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.606+0000 7f1fbeffd700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1fa80310e0 0x7f1fa80335a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1fb00096a0 tx=0x7f1fb0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:03.611 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.610+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1fb40079c0 con 0x7f1fc00f54b0 2024-01-11T05:11:03.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:03 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:02.343822+0000 mgr.smithi067.srbzpc (mgr.14156) 241 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:03.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.815+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc00fafa0 con 0x7f1fc00f54b0 2024-01-11T05:11:03.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.816+0000 7f1fa7fff700 1 -- 172.21.15.169:0/1511990012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fb40129d0 con 0x7f1fc00f54b0 2024-01-11T05:11:03.817 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:03.817 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.817+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1fa80310e0 msgr2=0x7f1fa80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1fa80310e0 0x7f1fa80335a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1fb00096a0 tx=0x7f1fb0006b40).stop 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 msgr2=0x7f1fc00fd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00fd490 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f1fb4004db0 tx=0x7f1fb400b040).stop 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 shutdown_connections 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f1fa80310e0 0x7f1fa80335a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 --2- 172.21.15.169:0/1511990012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fc00f54b0 0x7f1fc00fd490 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 >> 172.21.15.169:0/1511990012 conn(0x7f1fc00f0c10 msgr2=0x7f1fc005ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 shutdown_connections 2024-01-11T05:11:03.819 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:03.818+0000 7f1fc5aee700 1 -- 172.21.15.169:0/1511990012 wait complete. 2024-01-11T05:11:03.820 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:04.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:04 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:03.817281+0000 mon.smithi067 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.169:0/1511990012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:05.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:05.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:05.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:05 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:04.344075+0000 mgr.smithi067.srbzpc (mgr.14156) 242 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:06.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 -- 172.21.15.169:0/3708213551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 msgr2=0x7ffbe40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:06.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 --2- 172.21.15.169:0/3708213551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe40f58a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7ffbd4009fb0 tx=0x7ffbd4009b30).stop 2024-01-11T05:11:06.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 -- 172.21.15.169:0/3708213551 shutdown_connections 2024-01-11T05:11:06.303 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 --2- 172.21.15.169:0/3708213551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe40f58a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 -- 172.21.15.169:0/3708213551 >> 172.21.15.169:0/3708213551 conn(0x7ffbe40f0be0 msgr2=0x7ffbe40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 -- 172.21.15.169:0/3708213551 shutdown_connections 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.301+0000 7ffbea24b700 1 -- 172.21.15.169:0/3708213551 wait complete. 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.302+0000 7ffbea24b700 1 Processor -- start 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.302+0000 7ffbea24b700 1 -- start start 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.302+0000 7ffbea24b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:06.304 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.302+0000 7ffbea24b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe4060320 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.302+0000 7ffbe37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbe37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:57214/0 (socket says 172.21.15.169:57214) 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbe37fe700 1 -- 172.21.15.169:0/1368512842 learned_addr learned my addr 172.21.15.169:0/1368512842 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbe37fe700 1 -- 172.21.15.169:0/1368512842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd4005040 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbe37fe700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe41094f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ffbd40089f0 tx=0x7ffbd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbd4009320 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe4060520 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe4109ba0 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ffbd4013ec0 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.303+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbd4014d00 con 0x7ffbe40f5480 2024-01-11T05:11:06.306 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.304+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ffbd4014420 con 0x7ffbe40f5480 2024-01-11T05:11:06.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.304+0000 7ffbdbfff700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ffbd00310e0 0x7ffbd00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:06.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.304+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ffbd4042460 con 0x7ffbe40f5480 2024-01-11T05:11:06.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.304+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe4045bb0 con 0x7ffbe40f5480 2024-01-11T05:11:06.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.304+0000 7ffbe2ffd700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ffbd00310e0 0x7ffbd00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:06.307 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.305+0000 7ffbe2ffd700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ffbd00310e0 0x7ffbd00335a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ffbcc0096a0 tx=0x7ffbcc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:06.311 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.309+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ffbd4012b20 con 0x7ffbe40f5480 2024-01-11T05:11:06.518 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.517+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbe410a2f0 con 0x7ffbe40f5480 2024-01-11T05:11:06.519 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.518+0000 7ffbdbfff700 1 -- 172.21.15.169:0/1368512842 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffbd402b020 con 0x7ffbe40f5480 2024-01-11T05:11:06.519 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:06.519 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:06.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ffbd00310e0 msgr2=0x7ffbd00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:06.520 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ffbd00310e0 0x7ffbd00335a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ffbcc0096a0 tx=0x7ffbcc006b40).stop 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 msgr2=0x7ffbe41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe41094f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ffbd40089f0 tx=0x7ffbd400b040).stop 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 shutdown_connections 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7ffbd00310e0 0x7ffbd00335a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 --2- 172.21.15.169:0/1368512842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40f5480 0x7ffbe41094f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 >> 172.21.15.169:0/1368512842 conn(0x7ffbe40f0be0 msgr2=0x7ffbe40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 shutdown_connections 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:06.519+0000 7ffbea24b700 1 -- 172.21.15.169:0/1368512842 wait complete. 2024-01-11T05:11:06.521 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:07.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:06.344416+0000 mgr.smithi067.srbzpc (mgr.14156) 243 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:07.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:07 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:06.518903+0000 mon.smithi067 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.169:0/1368512842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:07.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:07.893 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/3263783500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 msgr2=0x7f6d980f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 --2- 172.21.15.169:0/3263783500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d980f69b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6d8c004760 tx=0x7f6d8c009ad0).stop 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/3263783500 shutdown_connections 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 --2- 172.21.15.169:0/3263783500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d980f69b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/3263783500 >> 172.21.15.169:0/3263783500 conn(0x7f6d980f1cd0 msgr2=0x7f6d980f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/3263783500 shutdown_connections 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.018+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/3263783500 wait complete. 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d9dc9c700 1 Processor -- start 2024-01-11T05:11:09.020 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d9dc9c700 1 -- start start 2024-01-11T05:11:09.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d9dc9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d981067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:09.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d9dc9c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d98106cb0 con 0x7f6d980f6590 2024-01-11T05:11:09.022 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d977fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d981067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d977fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d981067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:57236/0 (socket says 172.21.15.169:57236) 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.019+0000 7f6d977fe700 1 -- 172.21.15.169:0/1772784831 learned_addr learned my addr 172.21.15.169:0/1772784831 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d977fe700 1 -- 172.21.15.169:0/1772784831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c009d70 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d977fe700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d981067c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f6d8c004170 tx=0x7f6d8c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d8c00e040 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d98106eb0 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6d8c008ef0 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d8c014c90 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.020+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d98107340 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.021+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6d8c014420 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.021+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d98045bb0 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.021+0000 7f6d7ffff700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d80031090 0x7f6d80033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.021+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6d8c0421f0 con 0x7f6d980f6590 2024-01-11T05:11:09.023 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.021+0000 7f6d96ffd700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d80031090 0x7f6d80033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:09.024 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.022+0000 7f6d96ffd700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d80031090 0x7f6d80033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6d880096a0 tx=0x7f6d88006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:09.027 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.026+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6d8c026300 con 0x7f6d980f6590 2024-01-11T05:11:09.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.235+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d981081f0 con 0x7f6d980f6590 2024-01-11T05:11:09.237 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.236+0000 7f6d7ffff700 1 -- 172.21.15.169:0/1772784831 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d8c012b20 con 0x7f6d980f6590 2024-01-11T05:11:09.238 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:09.238 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d80031090 msgr2=0x7f6d80033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d80031090 0x7f6d80033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6d880096a0 tx=0x7f6d88006b40).stop 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 msgr2=0x7f6d981067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d981067c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f6d8c004170 tx=0x7f6d8c009ad0).stop 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 shutdown_connections 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f6d80031090 0x7f6d80033550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 --2- 172.21.15.169:0/1772784831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d980f6590 0x7f6d981067c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:09.239 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 >> 172.21.15.169:0/1772784831 conn(0x7f6d980f1cd0 msgr2=0x7f6d980f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:09.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 shutdown_connections 2024-01-11T05:11:09.240 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:09.238+0000 7f6d9dc9c700 1 -- 172.21.15.169:0/1772784831 wait complete. 2024-01-11T05:11:09.240 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:08.344765+0000 mgr.smithi067.srbzpc (mgr.14156) 244 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:09.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:09 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:09.237496+0000 mon.smithi067 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.169:0/1772784831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:10.606 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:10.606 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:11.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:11 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:10.345111+0000 mgr.smithi067.srbzpc (mgr.14156) 245 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:11.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 -- 172.21.15.169:0/3381344488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 msgr2=0x7fb4a00f4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:11.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 --2- 172.21.15.169:0/3381344488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f4ad0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb494009fb0 tx=0x7fb494009b30).stop 2024-01-11T05:11:11.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 -- 172.21.15.169:0/3381344488 shutdown_connections 2024-01-11T05:11:11.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 --2- 172.21.15.169:0/3381344488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f4ad0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:11.775 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 -- 172.21.15.169:0/3381344488 >> 172.21.15.169:0/3381344488 conn(0x7fb4a00f01d0 msgr2=0x7fb4a00f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:11.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 -- 172.21.15.169:0/3381344488 shutdown_connections 2024-01-11T05:11:11.776 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 -- 172.21.15.169:0/3381344488 wait complete. 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 Processor -- start 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.774+0000 7fb4a6b83700 1 -- start start 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a6b83700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a6b83700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00f56e0 con 0x7fb4a00f46b0 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a5b81700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a5b81700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45096/0 (socket says 172.21.15.169:45096) 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a5b81700 1 -- 172.21.15.169:0/2235151879 learned_addr learned my addr 172.21.15.169:0/2235151879 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a5b81700 1 -- 172.21.15.169:0/2235151879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb494005040 con 0x7fb4a00f46b0 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a5b81700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f7260 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb494000c00 tx=0x7fb49400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4940095f0 con 0x7fb4a00f46b0 2024-01-11T05:11:11.778 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb494007e50 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a00f5880 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb494014e70 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.775+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a00f5d10 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.776+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb494025070 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.776+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a00591e0 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.776+0000 7fb4927fc700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb48c031090 0x7fb48c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.776+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb494041dd0 con 0x7fb4a00f46b0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.777+0000 7fb4a5380700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb48c031090 0x7fb48c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:11.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.777+0000 7fb4a5380700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb48c031090 0x7fb48c033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb49c0096a0 tx=0x7fb49c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:11.782 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.782+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb49400f4e0 con 0x7fb4a00f46b0 2024-01-11T05:11:11.989 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.988+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4a00fbee0 con 0x7fb4a00f46b0 2024-01-11T05:11:11.990 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.988+0000 7fb4927fc700 1 -- 172.21.15.169:0/2235151879 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb49400f300 con 0x7fb4a00f46b0 2024-01-11T05:11:11.990 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:11.990 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:11.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb48c031090 msgr2=0x7fb48c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:11.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb48c031090 0x7fb48c033550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb49c0096a0 tx=0x7fb49c006b40).stop 2024-01-11T05:11:11.991 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 msgr2=0x7fb4a00f7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f7260 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb494000c00 tx=0x7fb49400b040).stop 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 shutdown_connections 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb48c031090 0x7fb48c033550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 --2- 172.21.15.169:0/2235151879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4a00f46b0 0x7fb4a00f7260 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 >> 172.21.15.169:0/2235151879 conn(0x7fb4a00f01d0 msgr2=0x7fb4a00f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 shutdown_connections 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:11.990+0000 7fb4a6b83700 1 -- 172.21.15.169:0/2235151879 wait complete. 2024-01-11T05:11:11.992 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:12.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:12 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:11.989987+0000 mon.smithi067 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.169:0/2235151879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:13.399 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:13.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:13.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:13 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:12.345425+0000 mgr.smithi067.srbzpc (mgr.14156) 246 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:14.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 -- 172.21.15.169:0/2828174067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 msgr2=0x7fba7c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:14.598 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 --2- 172.21.15.169:0/2828174067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c0f69b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fba6c004760 tx=0x7fba6c009ad0).stop 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 -- 172.21.15.169:0/2828174067 shutdown_connections 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 --2- 172.21.15.169:0/2828174067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c0f69b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 -- 172.21.15.169:0/2828174067 >> 172.21.15.169:0/2828174067 conn(0x7fba7c0f1cd0 msgr2=0x7fba7c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 -- 172.21.15.169:0/2828174067 shutdown_connections 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.596+0000 7fba83468700 1 -- 172.21.15.169:0/2828174067 wait complete. 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba83468700 1 Processor -- start 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba83468700 1 -- start start 2024-01-11T05:11:14.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba83468700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba83468700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c106cb0 con 0x7fba7c0f6590 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba81204700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba81204700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45118/0 (socket says 172.21.15.169:45118) 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.597+0000 7fba81204700 1 -- 172.21.15.169:0/1161140072 learned_addr learned my addr 172.21.15.169:0/1161140072 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba81204700 1 -- 172.21.15.169:0/1161140072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c009d70 con 0x7fba7c0f6590 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba81204700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c1067c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fba6c004170 tx=0x7fba6c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba6c00e040 con 0x7fba7c0f6590 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c106eb0 con 0x7fba7c0f6590 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fba6c008ef0 con 0x7fba7c0f6590 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba6c014c90 con 0x7fba7c0f6590 2024-01-11T05:11:14.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.598+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c107340 con 0x7fba7c0f6590 2024-01-11T05:11:14.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.599+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fba6c014420 con 0x7fba7c0f6590 2024-01-11T05:11:14.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.599+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c045bb0 con 0x7fba7c0f6590 2024-01-11T05:11:14.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.599+0000 7fba71ffb700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba680310e0 0x7fba680335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:14.601 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.599+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fba6c0422b0 con 0x7fba7c0f6590 2024-01-11T05:11:14.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.599+0000 7fba80a03700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba680310e0 0x7fba680335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:14.602 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.600+0000 7fba80a03700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba680310e0 0x7fba680335a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fba780096a0 tx=0x7fba78006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:14.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.604+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fba6c012d00 con 0x7fba7c0f6590 2024-01-11T05:11:14.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.814+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba7c1081f0 con 0x7fba7c0f6590 2024-01-11T05:11:14.814 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.814+0000 7fba71ffb700 1 -- 172.21.15.169:0/1161140072 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba6c012b20 con 0x7fba7c0f6590 2024-01-11T05:11:14.815 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:14.815 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:14.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba680310e0 msgr2=0x7fba680335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:14.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba680310e0 0x7fba680335a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fba780096a0 tx=0x7fba78006b40).stop 2024-01-11T05:11:14.816 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 msgr2=0x7fba7c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c1067c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fba6c004170 tx=0x7fba6c009ad0).stop 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 shutdown_connections 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fba680310e0 0x7fba680335a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 --2- 172.21.15.169:0/1161140072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7c0f6590 0x7fba7c1067c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 >> 172.21.15.169:0/1161140072 conn(0x7fba7c0f1cd0 msgr2=0x7fba7c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 shutdown_connections 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:14.816+0000 7fba83468700 1 -- 172.21.15.169:0/1161140072 wait complete. 2024-01-11T05:11:14.817 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:15.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:14.345788+0000 mgr.smithi067.srbzpc (mgr.14156) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:15.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:15 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:14.814796+0000 mon.smithi067 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.169:0/1161140072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:16.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:16.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:17.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/626672597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 msgr2=0x7fb1c40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 --2- 172.21.15.169:0/626672597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c40f58a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fb1c0009fb0 tx=0x7fb1c0009b30).stop 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/626672597 shutdown_connections 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 --2- 172.21.15.169:0/626672597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c40f58a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/626672597 >> 172.21.15.169:0/626672597 conn(0x7fb1c40f0be0 msgr2=0x7fb1c40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/626672597 shutdown_connections 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.350+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/626672597 wait complete. 2024-01-11T05:11:17.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1cc2c2700 1 Processor -- start 2024-01-11T05:11:17.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1cc2c2700 1 -- start start 2024-01-11T05:11:17.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1cc2c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c4108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:17.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1cc2c2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c4108ef0 con 0x7fb1c40f5480 2024-01-11T05:11:17.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1ca05e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c4108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:17.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1ca05e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c4108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45142/0 (socket says 172.21.15.169:45142) 2024-01-11T05:11:17.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.351+0000 7fb1ca05e700 1 -- 172.21.15.169:0/119770688 learned_addr learned my addr 172.21.15.169:0/119770688 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:17.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1ca05e700 1 -- 172.21.15.169:0/119770688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c0005040 con 0x7fb1c40f5480 2024-01-11T05:11:17.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1ca05e700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c4108a00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fb1c00089f0 tx=0x7fb1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:17.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1c0009320 con 0x7fb1c40f5480 2024-01-11T05:11:17.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c41090f0 con 0x7fb1c40f5480 2024-01-11T05:11:17.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb1c0013ec0 con 0x7fb1c40f5480 2024-01-11T05:11:17.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1c0014d30 con 0x7fb1c40f5480 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.352+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c4109580 con 0x7fb1c40f5480 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.353+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb1c00095f0 con 0x7fb1c40f5480 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.353+0000 7fb1baffd700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb1b0031090 0x7fb1b0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.353+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c4045bb0 con 0x7fb1c40f5480 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.353+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb1c00424a0 con 0x7fb1c40f5480 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.353+0000 7fb1c985d700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb1b0031090 0x7fb1b0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:17.357 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.354+0000 7fb1c985d700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb1b0031090 0x7fb1b0033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb1b40096a0 tx=0x7fb1b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:17.358 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.358+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb1c001d7f0 con 0x7fb1c40f5480 2024-01-11T05:11:17.569 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.569+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1c40fb060 con 0x7fb1c40f5480 2024-01-11T05:11:17.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.569+0000 7fb1baffd700 1 -- 172.21.15.169:0/119770688 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1c001d7f0 con 0x7fb1c40f5480 2024-01-11T05:11:17.570 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:17.570 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:17.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb1b0031090 msgr2=0x7fb1b0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:17.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb1b0031090 0x7fb1b0033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb1b40096a0 tx=0x7fb1b4006b40).stop 2024-01-11T05:11:17.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 msgr2=0x7fb1c4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:17.571 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c4108a00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fb1c00089f0 tx=0x7fb1c000b040).stop 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 shutdown_connections 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fb1b0031090 0x7fb1b0033550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 --2- 172.21.15.169:0/119770688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c40f5480 0x7fb1c4108a00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 >> 172.21.15.169:0/119770688 conn(0x7fb1c40f0be0 msgr2=0x7fb1c40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 shutdown_connections 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:17.571+0000 7fb1cc2c2700 1 -- 172.21.15.169:0/119770688 wait complete. 2024-01-11T05:11:17.572 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:17.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:17 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:16.346070+0000 mgr.smithi067.srbzpc (mgr.14156) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:18.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:18 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:17.569945+0000 mon.smithi067 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.169:0/119770688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:18.944 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:18.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:19.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:19 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:18.346345+0000 mgr.smithi067.srbzpc (mgr.14156) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:20.062 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 -- 172.21.15.169:0/1897414840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 msgr2=0x7fc8340f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:20.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 --2- 172.21.15.169:0/1897414840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc8340f58d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fc828009fb0 tx=0x7fc828009b30).stop 2024-01-11T05:11:20.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 -- 172.21.15.169:0/1897414840 shutdown_connections 2024-01-11T05:11:20.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 --2- 172.21.15.169:0/1897414840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc8340f58d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:20.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 -- 172.21.15.169:0/1897414840 >> 172.21.15.169:0/1897414840 conn(0x7fc8340f0c10 msgr2=0x7fc8340f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:20.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 -- 172.21.15.169:0/1897414840 shutdown_connections 2024-01-11T05:11:20.063 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.062+0000 7fc838f48700 1 -- 172.21.15.169:0/1897414840 wait complete. 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc838f48700 1 Processor -- start 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc838f48700 1 -- start start 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc838f48700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc834105700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc838f48700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834105bf0 con 0x7fc8340f54b0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc83259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc834105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc83259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc834105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35868/0 (socket says 172.21.15.169:35868) 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.063+0000 7fc83259c700 1 -- 172.21.15.169:0/2012971528 learned_addr learned my addr 172.21.15.169:0/2012971528 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc83259c700 1 -- 172.21.15.169:0/2012971528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828005040 con 0x7fc8340f54b0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc83259c700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc834105700 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fc8280089f0 tx=0x7fc82800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc828009320 con 0x7fc8340f54b0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc828013ec0 con 0x7fc8340f54b0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc834105df0 con 0x7fc8340f54b0 2024-01-11T05:11:20.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc828014d30 con 0x7fc8340f54b0 2024-01-11T05:11:20.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.064+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc834106280 con 0x7fc8340f54b0 2024-01-11T05:11:20.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.065+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc8280095f0 con 0x7fc8340f54b0 2024-01-11T05:11:20.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.065+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc834045bb0 con 0x7fc8340f54b0 2024-01-11T05:11:20.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.065+0000 7fc822ffd700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc81c031090 0x7fc81c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:20.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.065+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc8280423e0 con 0x7fc8340f54b0 2024-01-11T05:11:20.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.065+0000 7fc831d9b700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc81c031090 0x7fc81c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:20.068 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.066+0000 7fc831d9b700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc81c031090 0x7fc81c033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fc8240096a0 tx=0x7fc824006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:20.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.070+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc82801d980 con 0x7fc8340f54b0 2024-01-11T05:11:20.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.276+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8340fafa0 con 0x7fc8340f54b0 2024-01-11T05:11:20.276 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.276+0000 7fc822ffd700 1 -- 172.21.15.169:0/2012971528 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc82801d380 con 0x7fc8340f54b0 2024-01-11T05:11:20.276 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:20.277 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:20.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc81c031090 msgr2=0x7fc81c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:20.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc81c031090 0x7fc81c033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fc8240096a0 tx=0x7fc824006b40).stop 2024-01-11T05:11:20.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 msgr2=0x7fc834105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:20.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc834105700 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fc8280089f0 tx=0x7fc82800b040).stop 2024-01-11T05:11:20.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 shutdown_connections 2024-01-11T05:11:20.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fc81c031090 0x7fc81c033550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:20.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 --2- 172.21.15.169:0/2012971528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8340f54b0 0x7fc834105700 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:20.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 >> 172.21.15.169:0/2012971528 conn(0x7fc8340f0c10 msgr2=0x7fc8340f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:20.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 shutdown_connections 2024-01-11T05:11:20.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:20.278+0000 7fc838f48700 1 -- 172.21.15.169:0/2012971528 wait complete. 2024-01-11T05:11:20.279 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:20.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:20 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:20.276561+0000 mon.smithi067 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.169:0/2012971528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:21.639 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:21.639 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:21.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:21 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:20.346686+0000 mgr.smithi067.srbzpc (mgr.14156) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:22.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3495479502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 msgr2=0x7f10b40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:22.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 --2- 172.21.15.169:0/3495479502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b40f58a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f10a8009fb0 tx=0x7f10a8009b30).stop 2024-01-11T05:11:22.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3495479502 shutdown_connections 2024-01-11T05:11:22.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 --2- 172.21.15.169:0/3495479502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b40f58a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:22.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3495479502 >> 172.21.15.169:0/3495479502 conn(0x7f10b40f0be0 msgr2=0x7f10b40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:22.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3495479502 shutdown_connections 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.723+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3495479502 wait complete. 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10ba1e4700 1 Processor -- start 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10ba1e4700 1 -- start start 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10ba1e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b41094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10ba1e4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4060360 con 0x7f10b40f5480 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10b37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b41094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10b37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b41094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35886/0 (socket says 172.21.15.169:35886) 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10b37fe700 1 -- 172.21.15.169:0/3916223448 learned_addr learned my addr 172.21.15.169:0/3916223448 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.724+0000 7f10b37fe700 1 -- 172.21.15.169:0/3916223448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a8005040 con 0x7f10b40f5480 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.725+0000 7f10b37fe700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b41094b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f10a80089f0 tx=0x7f10a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.725+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10a8009320 con 0x7f10b40f5480 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.725+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f10a8013ec0 con 0x7f10b40f5480 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.725+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b4060560 con 0x7f10b40f5480 2024-01-11T05:11:22.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.725+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10a8014d30 con 0x7f10b40f5480 2024-01-11T05:11:22.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.725+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b4109be0 con 0x7f10b40f5480 2024-01-11T05:11:22.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.726+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f10a80095f0 con 0x7f10b40f5480 2024-01-11T05:11:22.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.726+0000 7f109bfff700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f109c031090 0x7f109c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:22.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.726+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4045bb0 con 0x7f10b40f5480 2024-01-11T05:11:22.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.726+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f10a80423e0 con 0x7f10b40f5480 2024-01-11T05:11:22.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.726+0000 7f10b2ffd700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f109c031090 0x7f109c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:22.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.727+0000 7f10b2ffd700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f109c031090 0x7f109c033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f10a40096a0 tx=0x7f10a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:22.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.731+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f10a801d7f0 con 0x7f10b40f5480 2024-01-11T05:11:22.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.941+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10b410a3d0 con 0x7f10b40f5480 2024-01-11T05:11:22.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.942+0000 7f109bfff700 1 -- 172.21.15.169:0/3916223448 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10a801d7f0 con 0x7f10b40f5480 2024-01-11T05:11:22.943 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:22.943 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:22.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f109c031090 msgr2=0x7f109c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f109c031090 0x7f109c033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f10a40096a0 tx=0x7f10a4006b40).stop 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 msgr2=0x7f10b41094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b41094b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f10a80089f0 tx=0x7f10a800b040).stop 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 shutdown_connections 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f109c031090 0x7f109c033550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 --2- 172.21.15.169:0/3916223448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40f5480 0x7f10b41094b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 >> 172.21.15.169:0/3916223448 conn(0x7f10b40f0be0 msgr2=0x7f10b40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 shutdown_connections 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:22.944+0000 7f10ba1e4700 1 -- 172.21.15.169:0/3916223448 wait complete. 2024-01-11T05:11:22.946 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:22.347075+0000 mgr.smithi067.srbzpc (mgr.14156) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:23.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:22.419810+0000 mon.smithi067 (mon.0) 301 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/trash_purge_schedule"}]: dispatch 2024-01-11T05:11:23.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:22.420081+0000 mon.smithi067 (mon.0) 302 : audit [INF] from='mgr.14156 172.21.15.67:0/1526956581' entity='mgr.smithi067.srbzpc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.srbzpc/mirror_snapshot_schedule"}]: dispatch 2024-01-11T05:11:23.763 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:23 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:22.943482+0000 mon.smithi067 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.169:0/3916223448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:24.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:24.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 -- 172.21.15.169:0/994968225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 msgr2=0x7f93ec0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 --2- 172.21.15.169:0/994968225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec0f58d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f93e0009fb0 tx=0x7f93e0009b30).stop 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 -- 172.21.15.169:0/994968225 shutdown_connections 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 --2- 172.21.15.169:0/994968225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec0f58d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 -- 172.21.15.169:0/994968225 >> 172.21.15.169:0/994968225 conn(0x7f93ec0f0c10 msgr2=0x7f93ec0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 -- 172.21.15.169:0/994968225 shutdown_connections 2024-01-11T05:11:25.466 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.464+0000 7f93f1916700 1 -- 172.21.15.169:0/994968225 wait complete. 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93f1916700 1 Processor -- start 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93f1916700 1 -- start start 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93f1916700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec10b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93f1916700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec107300 con 0x7f93ec0f54b0 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec10b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec10b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35910/0 (socket says 172.21.15.169:35910) 2024-01-11T05:11:25.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93eaffd700 1 -- 172.21.15.169:0/3338544743 learned_addr learned my addr 172.21.15.169:0/3338544743 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.465+0000 7f93eaffd700 1 -- 172.21.15.169:0/3338544743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e0005040 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.466+0000 7f93eaffd700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec10b910 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f93e00089f0 tx=0x7f93e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.466+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93e0009320 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.466+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec060300 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.466+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f93e0013ec0 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.466+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93e0014d30 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.466+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec10be00 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.467+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f93e0014420 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.467+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec045bb0 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.467+0000 7f93d37fe700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f93d40310e0 0x7f93d40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.467+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f93e0042470 con 0x7f93ec0f54b0 2024-01-11T05:11:25.469 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.467+0000 7f93ea7fc700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f93d40310e0 0x7f93d40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:25.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.468+0000 7f93ea7fc700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f93d40310e0 0x7f93d40335a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f93dc0096a0 tx=0x7f93dc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:25.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.472+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f93e001d980 con 0x7f93ec0f54b0 2024-01-11T05:11:25.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:25 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:24.347420+0000 mgr.smithi067.srbzpc (mgr.14156) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:25.682 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.681+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93ec0fafa0 con 0x7f93ec0f54b0 2024-01-11T05:11:25.683 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.682+0000 7f93d37fe700 1 -- 172.21.15.169:0/3338544743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93e001d380 con 0x7f93ec0f54b0 2024-01-11T05:11:25.683 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:25.683 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:25.684 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f93d40310e0 msgr2=0x7f93d40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f93d40310e0 0x7f93d40335a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f93dc0096a0 tx=0x7f93dc006b40).stop 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 msgr2=0x7f93ec10b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec10b910 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f93e00089f0 tx=0x7f93e000b040).stop 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 shutdown_connections 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f93d40310e0 0x7f93d40335a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 --2- 172.21.15.169:0/3338544743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec0f54b0 0x7f93ec10b910 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 >> 172.21.15.169:0/3338544743 conn(0x7f93ec0f0c10 msgr2=0x7f93ec0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 shutdown_connections 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:25.683+0000 7f93f1916700 1 -- 172.21.15.169:0/3338544743 wait complete. 2024-01-11T05:11:25.685 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:26.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:26 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:25.682918+0000 mon.smithi067 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.169:0/3338544743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:27.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:27.013 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:27.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:27 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:26.347787+0000 mgr.smithi067.srbzpc (mgr.14156) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:28.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 -- 172.21.15.169:0/2624437092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 msgr2=0x7fdd6c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:28.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 --2- 172.21.15.169:0/2624437092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c0f58a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fdd60009fb0 tx=0x7fdd60009b30).stop 2024-01-11T05:11:28.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 -- 172.21.15.169:0/2624437092 shutdown_connections 2024-01-11T05:11:28.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 --2- 172.21.15.169:0/2624437092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c0f58a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 -- 172.21.15.169:0/2624437092 >> 172.21.15.169:0/2624437092 conn(0x7fdd6c0f0be0 msgr2=0x7fdd6c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 -- 172.21.15.169:0/2624437092 shutdown_connections 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 -- 172.21.15.169:0/2624437092 wait complete. 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 Processor -- start 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.095+0000 7fdd731c0700 1 -- start start 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd731c0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd731c0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c060300 con 0x7fdd6c0f5480 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd70f5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd70f5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:35936/0 (socket says 172.21.15.169:35936) 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd70f5c700 1 -- 172.21.15.169:0/2016905092 learned_addr learned my addr 172.21.15.169:0/2016905092 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:28.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd70f5c700 1 -- 172.21.15.169:0/2016905092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60005040 con 0x7fdd6c0f5480 2024-01-11T05:11:28.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.096+0000 7fdd70f5c700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c109510 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fdd60000c00 tx=0x7fdd6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.097+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd600095f0 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.097+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c060500 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.097+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdd60007e50 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.097+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd60014e70 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.097+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c109bc0 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.097+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdd60025070 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.098+0000 7fdd697fa700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdd5c031090 0x7fdd5c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.098+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c0591e0 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.098+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdd60041d10 con 0x7fdd6c0f5480 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.098+0000 7fdd6bfff700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdd5c031090 0x7fdd5c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:28.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.098+0000 7fdd6bfff700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdd5c031090 0x7fdd5c033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fdd580096a0 tx=0x7fdd58006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:28.104 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.103+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdd600079c0 con 0x7fdd6c0f5480 2024-01-11T05:11:28.312 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.311+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd6c0faf70 con 0x7fdd6c0f5480 2024-01-11T05:11:28.313 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.311+0000 7fdd697fa700 1 -- 172.21.15.169:0/2016905092 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd6002d090 con 0x7fdd6c0f5480 2024-01-11T05:11:28.313 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:28.313 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:28.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdd5c031090 msgr2=0x7fdd5c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:28.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdd5c031090 0x7fdd5c033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fdd580096a0 tx=0x7fdd58006b40).stop 2024-01-11T05:11:28.314 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 msgr2=0x7fdd6c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c109510 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fdd60000c00 tx=0x7fdd6000b040).stop 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 shutdown_connections 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fdd5c031090 0x7fdd5c033550 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 --2- 172.21.15.169:0/2016905092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd6c0f5480 0x7fdd6c109510 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 >> 172.21.15.169:0/2016905092 conn(0x7fdd6c0f0be0 msgr2=0x7fdd6c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 shutdown_connections 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:28.313+0000 7fdd731c0700 1 -- 172.21.15.169:0/2016905092 wait complete. 2024-01-11T05:11:28.315 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:28.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:28 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:28.312861+0000 mon.smithi067 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.169:0/2016905092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:29.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:29.703 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:29.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:29 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:28.348089+0000 mgr.smithi067.srbzpc (mgr.14156) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:30.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- 172.21.15.169:0/685363525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 msgr2=0x7fbb400f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 --2- 172.21.15.169:0/685363525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb400f67f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fbb30009fb0 tx=0x7fbb30009b30).stop 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- 172.21.15.169:0/685363525 shutdown_connections 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 --2- 172.21.15.169:0/685363525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb400f67f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- 172.21.15.169:0/685363525 >> 172.21.15.169:0/685363525 conn(0x7fbb400f1b10 msgr2=0x7fbb400f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- 172.21.15.169:0/685363525 shutdown_connections 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- 172.21.15.169:0/685363525 wait complete. 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 Processor -- start 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- start start 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb40106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.851+0000 7fbb4688f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40106b30 con 0x7fbb400f63d0 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4588d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb40106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4588d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb40106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39662/0 (socket says 172.21.15.169:39662) 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4588d700 1 -- 172.21.15.169:0/1325293915 learned_addr learned my addr 172.21.15.169:0/1325293915 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:30.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4588d700 1 -- 172.21.15.169:0/1325293915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30005040 con 0x7fbb400f63d0 2024-01-11T05:11:30.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4588d700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb40106640 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fbb30000c00 tx=0x7fbb3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:30.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb300095f0 con 0x7fbb400f63d0 2024-01-11T05:11:30.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb40106d30 con 0x7fbb400f63d0 2024-01-11T05:11:30.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbb30007e50 con 0x7fbb400f63d0 2024-01-11T05:11:30.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb30014e70 con 0x7fbb400f63d0 2024-01-11T05:11:30.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.852+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb401071c0 con 0x7fbb400f63d0 2024-01-11T05:11:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.853+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbb300146b0 con 0x7fbb400f63d0 2024-01-11T05:11:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.853+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb40045bb0 con 0x7fbb400f63d0 2024-01-11T05:11:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.853+0000 7fbb367fc700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbb2c031090 0x7fbb2c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.853+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbb30041c50 con 0x7fbb400f63d0 2024-01-11T05:11:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.854+0000 7fbb4508c700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbb2c031090 0x7fbb2c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:30.856 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.854+0000 7fbb4508c700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbb2c031090 0x7fbb2c033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbb3c0096a0 tx=0x7fbb3c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:30.859 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:30.858+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbb3001b4a0 con 0x7fbb400f63d0 2024-01-11T05:11:31.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.066+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb40108030 con 0x7fbb400f63d0 2024-01-11T05:11:31.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.066+0000 7fbb367fc700 1 -- 172.21.15.169:0/1325293915 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb3001b2c0 con 0x7fbb400f63d0 2024-01-11T05:11:31.067 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:31.068 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:31.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbb2c031090 msgr2=0x7fbb2c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:31.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbb2c031090 0x7fbb2c033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbb3c0096a0 tx=0x7fbb3c006b40).stop 2024-01-11T05:11:31.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 msgr2=0x7fbb40106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:31.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb40106640 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fbb30000c00 tx=0x7fbb3000b040).stop 2024-01-11T05:11:31.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 shutdown_connections 2024-01-11T05:11:31.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fbb2c031090 0x7fbb2c033550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:31.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 --2- 172.21.15.169:0/1325293915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb400f63d0 0x7fbb40106640 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:31.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 >> 172.21.15.169:0/1325293915 conn(0x7fbb400f1b10 msgr2=0x7fbb400f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:31.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 shutdown_connections 2024-01-11T05:11:31.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:31.068+0000 7fbb4688f700 1 -- 172.21.15.169:0/1325293915 wait complete. 2024-01-11T05:11:31.070 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:30.348432+0000 mgr.smithi067.srbzpc (mgr.14156) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:31.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:31 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:31.067686+0000 mon.smithi067 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.169:0/1325293915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:32.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:32.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:33.590 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 -- 172.21.15.169:0/95813323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 msgr2=0x7f2aac0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 --2- 172.21.15.169:0/95813323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac0f58a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f2aa0009fb0 tx=0x7f2aa0009b30).stop 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 -- 172.21.15.169:0/95813323 shutdown_connections 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 --2- 172.21.15.169:0/95813323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac0f58a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 -- 172.21.15.169:0/95813323 >> 172.21.15.169:0/95813323 conn(0x7f2aac0f0be0 msgr2=0x7f2aac0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 -- 172.21.15.169:0/95813323 shutdown_connections 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.587+0000 7f2ab2e93700 1 -- 172.21.15.169:0/95813323 wait complete. 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab2e93700 1 Processor -- start 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab2e93700 1 -- start start 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab2e93700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab2e93700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac060320 con 0x7f2aac0f5480 2024-01-11T05:11:33.591 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab0c2f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab0c2f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39690/0 (socket says 172.21.15.169:39690) 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.588+0000 7f2ab0c2f700 1 -- 172.21.15.169:0/467114808 learned_addr learned my addr 172.21.15.169:0/467114808 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2ab0c2f700 1 -- 172.21.15.169:0/467114808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0005040 con 0x7f2aac0f5480 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2ab0c2f700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac1094f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f2aa00089f0 tx=0x7f2aa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa0009320 con 0x7f2aac0f5480 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac060520 con 0x7f2aac0f5480 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2aa0013ec0 con 0x7f2aac0f5480 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa0014d30 con 0x7f2aac0f5480 2024-01-11T05:11:33.592 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.589+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac109ba0 con 0x7f2aac0f5480 2024-01-11T05:11:33.593 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.590+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2aa00095f0 con 0x7f2aac0f5480 2024-01-11T05:11:33.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.590+0000 7f2aa97fa700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a9c0310e0 0x7f2a9c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:33.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.590+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2aa00424a0 con 0x7f2aac0f5480 2024-01-11T05:11:33.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.590+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac045bb0 con 0x7f2aac0f5480 2024-01-11T05:11:33.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.590+0000 7f2aabfff700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a9c0310e0 0x7f2a9c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:33.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.591+0000 7f2aabfff700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a9c0310e0 0x7f2a9c0335a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f2a980096a0 tx=0x7f2a98006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:33.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.595+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2aa001d7f0 con 0x7f2aac0f5480 2024-01-11T05:11:33.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:33 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:32.348860+0000 mgr.smithi067.srbzpc (mgr.14156) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:33.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.806+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2aac0fb060 con 0x7f2aac0f5480 2024-01-11T05:11:33.808 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.807+0000 7f2aa97fa700 1 -- 172.21.15.169:0/467114808 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2aa001d7f0 con 0x7f2aac0f5480 2024-01-11T05:11:33.809 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:33.809 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a9c0310e0 msgr2=0x7f2a9c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a9c0310e0 0x7f2a9c0335a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f2a980096a0 tx=0x7f2a98006b40).stop 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 msgr2=0x7f2aac1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac1094f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f2aa00089f0 tx=0x7f2aa000b040).stop 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 shutdown_connections 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2a9c0310e0 0x7f2a9c0335a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 --2- 172.21.15.169:0/467114808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aac0f5480 0x7f2aac1094f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 >> 172.21.15.169:0/467114808 conn(0x7f2aac0f0be0 msgr2=0x7f2aac0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 shutdown_connections 2024-01-11T05:11:33.810 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:33.809+0000 7f2ab2e93700 1 -- 172.21.15.169:0/467114808 wait complete. 2024-01-11T05:11:33.811 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:34.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:34 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:33.808446+0000 mon.smithi067 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.169:0/467114808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:35.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:35.176 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:35.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:35 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:34.349271+0000 mgr.smithi067.srbzpc (mgr.14156) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:36.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 -- 172.21.15.169:0/2617515190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 msgr2=0x7fa9800f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:36.335 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 --2- 172.21.15.169:0/2617515190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa9800f9160 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa974009fb0 tx=0x7fa974009b30).stop 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 -- 172.21.15.169:0/2617515190 shutdown_connections 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 --2- 172.21.15.169:0/2617515190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa9800f9160 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 -- 172.21.15.169:0/2617515190 >> 172.21.15.169:0/2617515190 conn(0x7fa9800f0be0 msgr2=0x7fa9800f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 -- 172.21.15.169:0/2617515190 shutdown_connections 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.333+0000 7fa985a97700 1 -- 172.21.15.169:0/2617515190 wait complete. 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa985a97700 1 Processor -- start 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa985a97700 1 -- start start 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa985a97700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa980106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa985a97700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa980106c40 con 0x7fa980060360 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa97f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa980106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa97f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa980106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39710/0 (socket says 172.21.15.169:39710) 2024-01-11T05:11:36.336 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.334+0000 7fa97f7fe700 1 -- 172.21.15.169:0/2483057566 learned_addr learned my addr 172.21.15.169:0/2483057566 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa97f7fe700 1 -- 172.21.15.169:0/2483057566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974005040 con 0x7fa980060360 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa97f7fe700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa980106750 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fa974000c00 tx=0x7fa97400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9740095f0 con 0x7fa980060360 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa980106e40 con 0x7fa980060360 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa974007e50 con 0x7fa980060360 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa974014e70 con 0x7fa980060360 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.335+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9801072d0 con 0x7fa980060360 2024-01-11T05:11:36.338 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.336+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa97401d4e0 con 0x7fa980060360 2024-01-11T05:11:36.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.336+0000 7fa967fff700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa968031090 0x7fa968033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:36.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.336+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa974041ca0 con 0x7fa980060360 2024-01-11T05:11:36.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.336+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9800591e0 con 0x7fa980060360 2024-01-11T05:11:36.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.336+0000 7fa97effd700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa968031090 0x7fa968033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:36.339 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.336+0000 7fa97effd700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa968031090 0x7fa968033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa9700096a0 tx=0x7fa970006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:36.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.341+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa97401b8b0 con 0x7fa980060360 2024-01-11T05:11:36.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.549+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9800fc060 con 0x7fa980060360 2024-01-11T05:11:36.551 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.550+0000 7fa967fff700 1 -- 172.21.15.169:0/2483057566 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa97402c090 con 0x7fa980060360 2024-01-11T05:11:36.551 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:36.551 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa968031090 msgr2=0x7fa968033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa968031090 0x7fa968033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa9700096a0 tx=0x7fa970006b40).stop 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 msgr2=0x7fa980106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa980106750 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fa974000c00 tx=0x7fa97400b040).stop 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 shutdown_connections 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fa968031090 0x7fa968033550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 --2- 172.21.15.169:0/2483057566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa980060360 0x7fa980106750 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.551+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 >> 172.21.15.169:0/2483057566 conn(0x7fa9800f0be0 msgr2=0x7fa9800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:36.553 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.552+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 shutdown_connections 2024-01-11T05:11:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:36.552+0000 7fa985a97700 1 -- 172.21.15.169:0/2483057566 wait complete. 2024-01-11T05:11:36.554 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:36.349718+0000 mgr.smithi067.srbzpc (mgr.14156) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:37.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:37 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:36.551243+0000 mon.smithi067 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.169:0/2483057566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:37.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:37.918 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 -- 172.21.15.169:0/2743611274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 msgr2=0x7f216c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 --2- 172.21.15.169:0/2743611274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c0f58a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f215c009fb0 tx=0x7f215c009b30).stop 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 -- 172.21.15.169:0/2743611274 shutdown_connections 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 --2- 172.21.15.169:0/2743611274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c0f58a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 -- 172.21.15.169:0/2743611274 >> 172.21.15.169:0/2743611274 conn(0x7f216c0f0be0 msgr2=0x7f216c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 -- 172.21.15.169:0/2743611274 shutdown_connections 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.992+0000 7f21736d6700 1 -- 172.21.15.169:0/2743611274 wait complete. 2024-01-11T05:11:38.995 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f21736d6700 1 Processor -- start 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f21736d6700 1 -- start start 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f21736d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f21736d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c106cd0 con 0x7f216c0f5480 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f2171472700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f2171472700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:39728/0 (socket says 172.21.15.169:39728) 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f2171472700 1 -- 172.21.15.169:0/3010513567 learned_addr learned my addr 172.21.15.169:0/3010513567 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f2171472700 1 -- 172.21.15.169:0/3010513567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c005040 con 0x7f216c0f5480 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.993+0000 7f2171472700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c1067e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f215c000c00 tx=0x7f215c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.994+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f215c0095f0 con 0x7f216c0f5480 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.994+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c106ed0 con 0x7f216c0f5480 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.994+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f215c007e50 con 0x7f216c0f5480 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.994+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f215c014e70 con 0x7f216c0f5480 2024-01-11T05:11:38.996 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.994+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c107360 con 0x7f216c0f5480 2024-01-11T05:11:38.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.995+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f215c025070 con 0x7f216c0f5480 2024-01-11T05:11:38.997 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.995+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c045bb0 con 0x7f216c0f5480 2024-01-11T05:11:38.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.995+0000 7f2161ffb700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2158031090 0x7f2158033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:38.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.995+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f215c041d90 con 0x7f216c0f5480 2024-01-11T05:11:38.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.995+0000 7f2170c71700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2158031090 0x7f2158033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:38.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:38.996+0000 7f2170c71700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2158031090 0x7f2158033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f21680096a0 tx=0x7f2168006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:39.001 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.000+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f215c00f4e0 con 0x7f216c0f5480 2024-01-11T05:11:39.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.207+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f216c108160 con 0x7f216c0f5480 2024-01-11T05:11:39.208 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.207+0000 7f2161ffb700 1 -- 172.21.15.169:0/3010513567 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f215c00f300 con 0x7f216c0f5480 2024-01-11T05:11:39.209 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:39.209 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:39.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2158031090 msgr2=0x7f2158033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:39.210 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2158031090 0x7f2158033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f21680096a0 tx=0x7f2168006b40).stop 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 msgr2=0x7f216c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c1067e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f215c000c00 tx=0x7f215c00b040).stop 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 shutdown_connections 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f2158031090 0x7f2158033550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 --2- 172.21.15.169:0/3010513567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f216c0f5480 0x7f216c1067e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 >> 172.21.15.169:0/3010513567 conn(0x7f216c0f0be0 msgr2=0x7f216c05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 shutdown_connections 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:39.209+0000 7f21736d6700 1 -- 172.21.15.169:0/3010513567 wait complete. 2024-01-11T05:11:39.211 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:39.704 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:38.350168+0000 mgr.smithi067.srbzpc (mgr.14156) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:39.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:39 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:39.208734+0000 mon.smithi067 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.169:0/3010513567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:40.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:40.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 -- 172.21.15.169:0/1990271091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 msgr2=0x7fd35c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 --2- 172.21.15.169:0/1990271091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c0f58a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fd344009fb0 tx=0x7fd344009b30).stop 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 -- 172.21.15.169:0/1990271091 shutdown_connections 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 --2- 172.21.15.169:0/1990271091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c0f58a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 -- 172.21.15.169:0/1990271091 >> 172.21.15.169:0/1990271091 conn(0x7fd35c0f0be0 msgr2=0x7fd35c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 -- 172.21.15.169:0/1990271091 shutdown_connections 2024-01-11T05:11:41.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.708+0000 7fd36088a700 1 -- 172.21.15.169:0/1990271091 wait complete. 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd36088a700 1 Processor -- start 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd36088a700 1 -- start start 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd36088a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd36088a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c060320 con 0x7fd35c0f5480 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd35a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd35a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42708/0 (socket says 172.21.15.169:42708) 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.709+0000 7fd35a59c700 1 -- 172.21.15.169:0/2989384837 learned_addr learned my addr 172.21.15.169:0/2989384837 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd35a59c700 1 -- 172.21.15.169:0/2989384837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344005040 con 0x7fd35c0f5480 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd35a59c700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c1094f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fd3440089f0 tx=0x7fd34400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd344009320 con 0x7fd35c0f5480 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c060520 con 0x7fd35c0f5480 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd344013ec0 con 0x7fd35c0f5480 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd344014d30 con 0x7fd35c0f5480 2024-01-11T05:11:41.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.710+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c109ba0 con 0x7fd35c0f5480 2024-01-11T05:11:41.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.711+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd3440095f0 con 0x7fd35c0f5480 2024-01-11T05:11:41.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.711+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c045bb0 con 0x7fd35c0f5480 2024-01-11T05:11:41.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.711+0000 7fd352ffd700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd348031090 0x7fd348033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:41.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.711+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd3440423e0 con 0x7fd35c0f5480 2024-01-11T05:11:41.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.711+0000 7fd359d9b700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd348031090 0x7fd348033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:41.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.712+0000 7fd359d9b700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd348031090 0x7fd348033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd34c0096a0 tx=0x7fd34c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:41.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.716+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd34401d980 con 0x7fd35c0f5480 2024-01-11T05:11:41.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:41 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:40.350607+0000 mgr.smithi067.srbzpc (mgr.14156) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:41.923 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.922+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd35c0faf70 con 0x7fd35c0f5480 2024-01-11T05:11:41.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.923+0000 7fd352ffd700 1 -- 172.21.15.169:0/2989384837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd34401d380 con 0x7fd35c0f5480 2024-01-11T05:11:41.924 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:41.924 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd348031090 msgr2=0x7fd348033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd348031090 0x7fd348033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd34c0096a0 tx=0x7fd34c006b40).stop 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 msgr2=0x7fd35c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c1094f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fd3440089f0 tx=0x7fd34400b040).stop 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 shutdown_connections 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7fd348031090 0x7fd348033550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 --2- 172.21.15.169:0/2989384837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35c0f5480 0x7fd35c1094f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 >> 172.21.15.169:0/2989384837 conn(0x7fd35c0f0be0 msgr2=0x7fd35c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 shutdown_connections 2024-01-11T05:11:41.926 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:41.924+0000 7fd36088a700 1 -- 172.21.15.169:0/2989384837 wait complete. 2024-01-11T05:11:41.927 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:42.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:42 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:41.923902+0000 mon.smithi067 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.169:0/2989384837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:43.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:43.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:43.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:43 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:42.350987+0000 mgr.smithi067.srbzpc (mgr.14156) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 -- 172.21.15.169:0/1319239251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 msgr2=0x7f42900f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 --2- 172.21.15.169:0/1319239251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42900f58d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f4284009fb0 tx=0x7f4284009b30).stop 2024-01-11T05:11:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 -- 172.21.15.169:0/1319239251 shutdown_connections 2024-01-11T05:11:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 --2- 172.21.15.169:0/1319239251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42900f58d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 -- 172.21.15.169:0/1319239251 >> 172.21.15.169:0/1319239251 conn(0x7f42900f0c10 msgr2=0x7f42900f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:44.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 -- 172.21.15.169:0/1319239251 shutdown_connections 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.409+0000 7f4296211700 1 -- 172.21.15.169:0/1319239251 wait complete. 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f4296211700 1 Processor -- start 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f4296211700 1 -- start start 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f4296211700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42901094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f4296211700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4290060320 con 0x7f42900f54b0 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42901094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:44.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42901094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42726/0 (socket says 172.21.15.169:42726) 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.410+0000 7f428f7fe700 1 -- 172.21.15.169:0/3802243017 learned_addr learned my addr 172.21.15.169:0/3802243017 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f428f7fe700 1 -- 172.21.15.169:0/3802243017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4284005040 con 0x7f42900f54b0 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f428f7fe700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42901094a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f42840089f0 tx=0x7f428400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4284009320 con 0x7f42900f54b0 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4290060520 con 0x7f42900f54b0 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4284013ec0 con 0x7f42900f54b0 2024-01-11T05:11:44.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4284014d30 con 0x7f42900f54b0 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.411+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4290109b50 con 0x7f42900f54b0 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.412+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4284014420 con 0x7f42900f54b0 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.412+0000 7f4277fff700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4278031090 0x7f4278033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.412+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4290045bb0 con 0x7f42900f54b0 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.412+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f42840424c0 con 0x7f42900f54b0 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.412+0000 7f428effd700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4278031090 0x7f4278033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:44.417 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.413+0000 7f428effd700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4278031090 0x7f4278033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f42800096a0 tx=0x7f4280006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:44.418 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.417+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f428401d980 con 0x7f42900f54b0 2024-01-11T05:11:44.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.623+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f429010a370 con 0x7f42900f54b0 2024-01-11T05:11:44.624 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.623+0000 7f4277fff700 1 -- 172.21.15.169:0/3802243017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f428401d380 con 0x7f42900f54b0 2024-01-11T05:11:44.625 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:44.625 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:44.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4278031090 msgr2=0x7f4278033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:44.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4278031090 0x7f4278033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f42800096a0 tx=0x7f4280006b40).stop 2024-01-11T05:11:44.626 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 msgr2=0x7f42901094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42901094a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f42840089f0 tx=0x7f428400b040).stop 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 shutdown_connections 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f4278031090 0x7f4278033550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 --2- 172.21.15.169:0/3802243017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42900f54b0 0x7f42901094a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 >> 172.21.15.169:0/3802243017 conn(0x7f42900f0c10 msgr2=0x7f42900f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 shutdown_connections 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:44.625+0000 7f4296211700 1 -- 172.21.15.169:0/3802243017 wait complete. 2024-01-11T05:11:44.627 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:44.351296+0000 mgr.smithi067.srbzpc (mgr.14156) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:45.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:45 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:44.624780+0000 mon.smithi067 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.169:0/3802243017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:46.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:46.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:47.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 -- 172.21.15.169:0/2207779117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 msgr2=0x7f80b40f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:47.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 --2- 172.21.15.169:0/2207779117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b40f5020 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f80b0009fb0 tx=0x7f80b0009b30).stop 2024-01-11T05:11:47.134 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 -- 172.21.15.169:0/2207779117 shutdown_connections 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 --2- 172.21.15.169:0/2207779117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b40f5020 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 -- 172.21.15.169:0/2207779117 >> 172.21.15.169:0/2207779117 conn(0x7f80b40f0360 msgr2=0x7f80b40f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 -- 172.21.15.169:0/2207779117 shutdown_connections 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.132+0000 7f80bbc4c700 1 -- 172.21.15.169:0/2207779117 wait complete. 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80bbc4c700 1 Processor -- start 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80bbc4c700 1 -- start start 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80bbc4c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b4106d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80bbc4c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b4107270 con 0x7f80b40f4c00 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80b99e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b4106d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80b99e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b4106d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:42746/0 (socket says 172.21.15.169:42746) 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80b99e8700 1 -- 172.21.15.169:0/3655384662 learned_addr learned my addr 172.21.15.169:0/3655384662 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:47.135 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.133+0000 7f80b99e8700 1 -- 172.21.15.169:0/3655384662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b0005040 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.134+0000 7f80b99e8700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b4106d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f80b00089f0 tx=0x7f80b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.134+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b0009320 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.134+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b4107470 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.134+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f80b0013ec0 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.134+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b0014d30 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.134+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b4107900 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.135+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f80b00095f0 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.135+0000 7f80aa7fc700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80a0031090 0x7f80a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.135+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b4045bb0 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.135+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f80b00423e0 con 0x7f80b40f4c00 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.135+0000 7f80b91e7700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80a0031090 0x7f80a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:47.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.135+0000 7f80b91e7700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80a0031090 0x7f80a0033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f80a40096a0 tx=0x7f80a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:47.141 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.140+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f80b000fe40 con 0x7f80b40f4c00 2024-01-11T05:11:47.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.345+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80b40fa6f0 con 0x7f80b40f4c00 2024-01-11T05:11:47.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.346+0000 7f80aa7fc700 1 -- 172.21.15.169:0/3655384662 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80b000fe40 con 0x7f80b40f4c00 2024-01-11T05:11:47.347 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:47.347 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80a0031090 msgr2=0x7f80a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80a0031090 0x7f80a0033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f80a40096a0 tx=0x7f80a4006b40).stop 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 msgr2=0x7f80b4106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b4106d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f80b00089f0 tx=0x7f80b000b040).stop 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 shutdown_connections 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f80a0031090 0x7f80a0033550 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 --2- 172.21.15.169:0/3655384662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b40f4c00 0x7f80b4106d80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.347+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 >> 172.21.15.169:0/3655384662 conn(0x7f80b40f0360 msgr2=0x7f80b40f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.348+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 shutdown_connections 2024-01-11T05:11:47.349 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:47.348+0000 7f80bbc4c700 1 -- 172.21.15.169:0/3655384662 wait complete. 2024-01-11T05:11:47.350 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:46.351636+0000 mgr.smithi067.srbzpc (mgr.14156) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:47.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:47 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:47.347266+0000 mon.smithi067 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.169:0/3655384662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:48.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-11T05:11:48.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 -- ceph mon dump -f json 2024-01-11T05:11:49.762 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:49 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: cluster 2024-01-11T05:11:48.351970+0000 mgr.smithi067.srbzpc (mgr.14156) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-11T05:11:49.851 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.848+0000 7f8c2cd0e700 1 -- 172.21.15.169:0/1004152279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c18013860 con 0x7f8c280f5480 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 -- 172.21.15.169:0/1004152279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 msgr2=0x7f8c280f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 --2- 172.21.15.169:0/1004152279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c280f58a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f8c18009fb0 tx=0x7f8c18009b30).stop 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 -- 172.21.15.169:0/1004152279 shutdown_connections 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 --2- 172.21.15.169:0/1004152279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c280f58a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 -- 172.21.15.169:0/1004152279 >> 172.21.15.169:0/1004152279 conn(0x7f8c280f0be0 msgr2=0x7f8c280f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 -- 172.21.15.169:0/1004152279 shutdown_connections 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.849+0000 7f8c2ff74700 1 -- 172.21.15.169:0/1004152279 wait complete. 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2ff74700 1 Processor -- start 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2ff74700 1 -- start start 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2ff74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c281067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2ff74700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28106ce0 con 0x7f8c280f5480 2024-01-11T05:11:49.852 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2dd10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c281067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:49.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2dd10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c281067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.169:45022/0 (socket says 172.21.15.169:45022) 2024-01-11T05:11:49.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2dd10700 1 -- 172.21.15.169:0/2184924267 learned_addr learned my addr 172.21.15.169:0/2184924267 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-01-11T05:11:49.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.850+0000 7f8c2dd10700 1 -- 172.21.15.169:0/2184924267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18005040 con 0x7f8c280f5480 2024-01-11T05:11:49.853 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.851+0000 7f8c2dd10700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c281067f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f8c180089f0 tx=0x7f8c1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.851+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c18009320 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.851+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c28106ee0 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.851+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8c18013ec0 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.851+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c18014d30 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.851+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c281072f0 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.852+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8c18014420 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.852+0000 7f8c1e7fc700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c140354a0 0x7f8c14037960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.852+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c28045bb0 con 0x7f8c280f5480 2024-01-11T05:11:49.854 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.852+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8c180423e0 con 0x7f8c280f5480 2024-01-11T05:11:49.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.852+0000 7f8c2d50f700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c140354a0 0x7f8c14037960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-11T05:11:49.855 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.852+0000 7f8c2d50f700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c140354a0 0x7f8c14037960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8c240096a0 tx=0x7f8c24006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-11T05:11:49.858 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:49.857+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8c1801d7f0 con 0x7f8c280f5480 2024-01-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.065+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c280fb060 con 0x7f8c280f5480 2024-01-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.066+0000 7f8c1e7fc700 1 -- 172.21.15.169:0/2184924267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c1801d7f0 con 0x7f8c280f5480 2024-01-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-01-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"89773ea0-b03e-11ee-95ab-87774f69a715","modified":"2024-01-11T05:02:33.494211Z","created":"2024-01-11T05:02:33.494211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c140354a0 msgr2=0x7f8c14037960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c140354a0 0x7f8c14037960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8c240096a0 tx=0x7f8c24006b40).stop 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 msgr2=0x7f8c281067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c281067f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f8c180089f0 tx=0x7f8c1800b040).stop 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 shutdown_connections 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:6800/1409268088,v1:172.21.15.67:6801/1409268088] conn(0x7f8c140354a0 0x7f8c14037960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 --2- 172.21.15.169:0/2184924267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280f5480 0x7f8c281067f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-11T05:11:50.069 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 >> 172.21.15.169:0/2184924267 conn(0x7f8c280f0be0 msgr2=0x7f8c280f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-11T05:11:50.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 shutdown_connections 2024-01-11T05:11:50.070 INFO:teuthology.orchestra.run.smithi169.stderr:2024-01-11T05:11:50.067+0000 7f8c2ff74700 1 -- 172.21.15.169:0/2184924267 wait complete. 2024-01-11T05:11:50.070 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-01-11T05:11:50.443 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-11T05:11:50.444 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-11T05:11:50.444 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-11T05:11:50.469 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-11T05:11:50.487 INFO:tasks.cephadm:Stopping all daemons... 2024-01-11T05:11:50.487 INFO:tasks.cephadm.mon.smithi067:Stopping mon.smithi067... 2024-01-11T05:11:50.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067 2024-01-11T05:11:50.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: audit 2024-01-11T05:11:50.067164+0000 mon.smithi067 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.169:0/2184924267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-11T05:11:50.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:50 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for 89773ea0-b03e-11ee-95ab-87774f69a715... 2024-01-11T05:11:50.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:50 smithi067 podman[44916]: Error: no container with name or ID "ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon.smithi067" found: no such container 2024-01-11T05:11:50.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:11:50.747+0000 7fee94abe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-11T05:11:50.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:50 smithi067 ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067[38928]: debug 2024-01-11T05:11:50.747+0000 7fee94abe700 -1 mon.smithi067@0(leader) e1 *** Got Signal Terminated *** 2024-01-11T05:11:51.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Jan 11 05:11:50 smithi067 bash[44957]: ceph-89773ea0-b03e-11ee-95ab-87774f69a715-mon-smithi067 2024-01-11T05:11:51.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi067.service' 2024-01-11T05:11:51.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-11T05:11:51.111 INFO:tasks.cephadm.mon.smithi067:Stopped mon.smithi067 2024-01-11T05:11:51.112 INFO:tasks.cephadm.mon.smithi169:Stopping mon.smithi169... 2024-01-11T05:11:51.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169 2024-01-11T05:11:51.146 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169.service: Unit ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169.service not loaded. 2024-01-11T05:11:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-11T05:11:51.148 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi169" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 5: 'sudo systemctl stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169' 2024-01-11T05:11:51.149 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-11T05:11:51.151 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/remote/smithi067/crash 2024-01-11T05:11:51.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/crash -- . 2024-01-11T05:11:51.193 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/remote/smithi169/crash 2024-01-11T05:11:51.194 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/crash -- . 2024-01-11T05:11:51.242 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/crash: Cannot open: No such file or directory 2024-01-11T05:11:51.242 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2024-01-11T05:11:51.243 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-11T05:11:51.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-11T05:11:51.274 INFO:teuthology.orchestra.run.smithi067.stderr:grep: /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph.log: No such file or directory 2024-01-11T05:11:51.276 INFO:tasks.cephadm:Compressing logs... 2024-01-11T05:11:51.276 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-11T05:11:51.318 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-11T05:11:51.343 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-11T05:11:51.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-11T05:11:51.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-volume.log 2024-01-11T05:11:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mgr.smithi067.srbzpc.log 2024-01-11T05:11:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mon.smithi067.log 2024-01-11T05:11:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-volume.log: /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mgr.smithi067.srbzpc.log: /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mon.smithi067.log: 90.2% -- replaced with /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-volume.log.gz 2024-01-11T05:11:51.346 INFO:teuthology.orchestra.run.smithi169.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-11T05:11:51.346 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-11T05:11:51.349 INFO:teuthology.orchestra.run.smithi067.stderr: 92.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-11T05:11:51.351 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-11T05:11:51.352 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-01-11T05:11:51.352 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m0.016s 2024-01-11T05:11:51.352 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m0.012s 2024-01-11T05:11:51.353 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.013s 2024-01-11T05:11:51.416 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mgr.smithi067.srbzpc.log: file size changed while zipping 2024-01-11T05:11:51.416 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mgr.smithi067.srbzpc.log.gz 2024-01-11T05:11:51.487 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/89773ea0-b03e-11ee-95ab-87774f69a715/ceph-mon.smithi067.log.gz 2024-01-11T05:11:51.489 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-01-11T05:11:51.489 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m0.156s 2024-01-11T05:11:51.489 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m0.222s 2024-01-11T05:11:51.489 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.022s 2024-01-11T05:11:51.489 INFO:tasks.cephadm:Archiving logs... 2024-01-11T05:11:51.490 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/remote/smithi067/log 2024-01-11T05:11:51.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-11T05:11:51.603 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/remote/smithi169/log 2024-01-11T05:11:51.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-11T05:11:51.638 INFO:tasks.cephadm:Removing cluster... 2024-01-11T05:11:51.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 --force 2024-01-11T05:13:23.442 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 89773ea0-b03e-11ee-95ab-87774f69a715 --force 2024-01-11T05:13:23.896 INFO:tasks.cephadm:Removing cephadm ... 2024-01-11T05:13:23.897 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-11T05:13:23.918 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-11T05:13:23.936 INFO:tasks.cephadm:Teardown complete 2024-01-11T05:13:23.936 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 5: 'sudo systemctl stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169' 2024-01-11T05:13:24.168 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=103d7ccee0a9441e87a8581c888b8300 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_4bcedb9fc11bed1fd6fd0c4fd59187401f309680/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 5: 'sudo systemctl stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169' 2024-01-11T05:13:24.173 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-11T05:13:24.188 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-11T05:13:24.225 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-11T05:13:24.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-11T05:13:24.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-11T05:13:24.329 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-11T05:13:24.338 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-11T05:13:24.338 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T05:13:24.341 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T05:13:24.358 INFO:teuthology.orchestra.run.smithi169.stderr:bash: ntpq: command not found 2024-01-11T05:13:24.359 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2024-01-11T05:13:24.363 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T05:13:24.363 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-01-11T05:13:24.363 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 -1139us[-1138us] +/- 86ms 2024-01-11T05:13:24.364 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -132us[ -131us] +/- 76ms 2024-01-11T05:13:24.364 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +1303us[+1304us] +/- 84ms 2024-01-11T05:13:24.364 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T05:13:24.364 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T05:13:24.364 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-01-11T05:13:24.365 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 -1151us[-1085us] +/- 86ms 2024-01-11T05:13:24.365 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -139us[ -73us] +/- 76ms 2024-01-11T05:13:24.365 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +1302us[+1368us] +/- 84ms 2024-01-11T05:13:24.365 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T05:13:24.366 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-11T05:13:24.375 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-11T05:13:24.376 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-11T05:13:24.399 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-11T05:13:24.476 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-11T05:13:24.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T05:13:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T05:13:24.573 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T05:13:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T05:13:24.603 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-11T05:13:24.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-11T05:13:24.658 INFO:teuthology.task.internal:Duration was 1279.199175 seconds 2024-01-11T05:13:24.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-11T05:13:24.691 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-11T05:13:24.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-11T05:13:24.694 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-11T05:13:24.741 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T05:13:24.743 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T05:13:25.026 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-11T05:13:25.026 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-01-11T05:13:25.027 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-11T05:13:25.052 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-01-11T05:13:25.053 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-11T05:13:25.075 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-11T05:13:25.075 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-11T05:13:25.095 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-11T05:13:25.164 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-11T05:13:25.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-11T05:13:25.166 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-11T05:13:25.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-11T05:13:25.266 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-11T05:13:25.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-11T05:13:25.294 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-11T05:13:25.321 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-11T05:13:25.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-11T05:13:25.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-11T05:13:25.361 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-01-11T05:13:25.387 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-01-11T05:13:25.446 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-11T05:13:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T05:13:25.464 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-11T05:13:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T05:13:25.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-11T05:13:25.514 INFO:teuthology.task.internal:Transferring archived files... 2024-01-11T05:13:25.515 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/remote/smithi067 2024-01-11T05:13:25.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-11T05:13:25.556 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279/remote/smithi169 2024-01-11T05:13:25.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-11T05:13:25.603 INFO:teuthology.task.internal:Removing archive directory... 2024-01-11T05:13:25.603 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-11T05:13:25.606 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-11T05:13:25.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-11T05:13:25.666 INFO:teuthology.task.internal:Not uploading archives. 2024-01-11T05:13:25.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-11T05:13:25.683 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-11T05:13:25.683 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-11T05:13:25.686 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-11T05:13:25.701 INFO:teuthology.orchestra.run.smithi067.stdout: 526044 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 11 05:13 /home/ubuntu/cephtest 2024-01-11T05:13:25.710 INFO:teuthology.orchestra.run.smithi169.stdout: 525359 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 11 05:13 /home/ubuntu/cephtest 2024-01-11T05:13:25.711 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-11T05:13:25.722 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-11T05:13:25.785 INFO:teuthology.nuke:Checking targets against current locks 2024-01-11T05:13:25.811 DEBUG:teuthology.nuke:shortname: smithi067 2024-01-11T05:13:25.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T05:13:25.837 DEBUG:teuthology.nuke:shortname: smithi169 2024-01-11T05:13:25.837 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T05:13:25.841 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 04:43:46.658862', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNh5DBjwgYTSwL/3YloYZJSjDOD+LxmDNsWW17id/sGjVny/f2XlzmuPkW+zwV6hQbcSkEIKHIPSFEzoPJI+QRM='} 2024-01-11T05:13:25.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-10_19:20:36-fs-wip-vshankar-testing1-quincy-2024-01-10-2010-quincy-distro-default-smithi/7512279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 04:43:46.656854', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLJxRwirZkrqx5TImKRLdwq1OP0U4I4EXXu+V8JheRl7rQwhiWlPFk5nAsBGmKYad7QJ2R5mPWpMHZQHN+mgit8='} 2024-01-11T05:13:25.892 INFO:teuthology.orchestra.console.smithi067:Power off 2024-01-11T05:13:25.892 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-11T05:13:25.916 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-01-11T05:13:25.916 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T05:13:25.935 INFO:teuthology.orchestra.console.smithi169:Power off 2024-01-11T05:13:25.936 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-11T05:13:25.945 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-01-11T05:13:25.958 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-01-11T05:13:25.958 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T05:13:25.986 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-01-11T05:13:29.947 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T05:13:29.988 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T05:13:30.058 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-01-11T05:13:30.100 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-01-11T05:13:34.063 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T05:13:34.100 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T05:13:34.174 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-01-11T05:13:34.211 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-01-11T05:13:34.275 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-01-11T05:13:34.313 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-01-11T05:13:34.418 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1279.1991748809814 failure_reason: 'Command failed on smithi169 with status 5: ''sudo systemctl stop ceph-89773ea0-b03e-11ee-95ab-87774f69a715@mon.smithi169''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=103d7ccee0a9441e87a8581c888b8300 status: fail success: false 2024-01-11T05:13:34.418 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T05:13:34.498 INFO:teuthology.run:FAIL